CD-EXCALIBUR-FMS0082-M4c.4_ExcaliburFusionModellingSystemProgressTowardsImplementation3DTurbulenceCodeTight
===========================================================================================================

.. meta::
   :description: technical note

   :keywords: ExCALIBUR,3-D,integrated,particle,and,continuum,model,M4c.4,Version,1.00,Abstract,This,report,describes,work,for,ExCALIBUR,project,NEPTUNE,at,Mile-,stone,M4c.4.,Further,developments,have,been,made,to,the,integrated,particle,and,con-,tinuum,model,in,“3-D,3-V”,,where,3-D,3-V,implies,that,there,is,variation,in,three,space,dimensions,(3-D),and,three,velocity,space,coordinates,(3-,V).,Nektar++,provides,the,3-D,finite,elements,to,represent,the,plasma,as,a,fluid,,whereas,3-D,3-V,phase,space,is,populated,with,neutral,particles,by,the,SYCL-enabled,NESO-Particles,library.,The,calculation,is,orches-,trated,by,the,NESO,software,,which,has,been,developed,under,project,NEPTUNE.,Work,reported,herein,includes:,the,ParticleLoop,implemen-,tation,in,NESO-Particles;,the,parallel,dynamics,of,the,Hasegawa-Wakatani,model;,and,the,recombination,reaction,kernel,for,neutral,particles.,UKAEA,REFERENCE,AND,APPROVAL,SHEET,Client,Reference:,UKAEA,Reference:,CD/EXCALIBUR-FMS/0082,Issue:,Date:,1.00,March,15,,2024,Project,Name:,ExCALIBUR,Fusion,Modelling,System,Prepared,By:,Name,and,Department,Will,Saunders,James,Cook,Signature,N/A,N/A,Date,March,15,,2024,March,15,,2024,CD,Reviewed,By:,Ed,Threlfall,March,15,,2024,Deputy,Principal,Investiga-,tor,1,1,Introduction,In,this,report,we,describe,progress,made,towards,the,implementation,of,a,3-D,turbulence,code,tightly,coupled,to,kinetic,neutrals.,We,discuss,the,augmentation,of,the,implementation,of,the,Hasegawa-Wakatani,equations,such,that,the,parallel,dynamics,are,accurately,incorporated,into,tightly,coupled,simulations,of,turbulence,with,newly,developed,neutral,physics.,Particles,interact,with,the,finite-element,fluid,via,an,ionisation,source,term,and,its,reciprocal,action,,recombination.,The,particle,density,is,coupled,to,the,plasma,density,by,a,time,varying,source,term,determined,by,the,spatio-temporal,feedback,between,the,neutral,particles,and,the,plasma.,This,report,describes,the,following,developments:,upgrades,to,NESO-Particles,to,use,a,ParticleLoop,,amongst,other,improvements;,a,recombination,kernel,in,NESO-Particles.,2,2,Task,Work,2.1,NESO-Particles,Additions,2.1.1,ParticleLoop,A,“particle,loop”,is,a,looping,operation,where,a,single,“kernel”,is,applied,to,all,particles,in,a,partic-,ular,group,of,particles,[1].,Previously,we,described,a,high,level,particle,loop,interface,embedded,within,the,Julia,programming,language,[2].,This,Julia,implementation,closely,followed,the,original,PPMD,implementation,with,the,computational,kernel,described,by,the,user,as,a,string,literal.,This,string,literal,approach,is,not,suitable,for,the,SYCL,(C++),NESO-Particles,(NP),implementation,of,the,abstraction,as,it,adds,additional,complexity,to,consume,and,manipulate,the,string,to,create,a,kernel,that,can,be,passed,to,the,SYCL,implementation.,The,initial,implementation,of,NP,required,that,users,write,a,particle,loop,as,a,SYCL,parallel,loop.,An,example,of,such,a,loop,is,presented,in,Listing,1.,This,initial,implementation,has,several,short-,comings,,it,requires,the,user,to,understand,the,SYCL,programming,model,and,directly,interface,It,requires,a,user,to,write,excessive,“boilerplate”,code,to,with,particle,data,via,device,pointers.,perform,relatively,simple,operations.,It,is,relatively,easy,to,write,code,that,is,not,correct,and,is,hard,to,visually,inspect,for,the,identification,of,issues.,The,code,is,relatively,inflexible,,e.g.,exactly,how,the,particle,loop,is,executed,is,difficult,to,adjust.,Listing,1:,Define,and,launch,an,old,style,particle,loop,to,apply,a,Forward,Euler,like,operation.,auto,k_P,=,(*,particle_group,)[,Sym,<,REAL,>(,",POSITION,",)],->,cell_dat,.,device_ptr,();,auto,k_V,=,(*,particle_group,)[,Sym,<,REAL,>(,",VELOCITY,",)],->,cell_dat,.,device_ptr,();,auto,pl_iter_range,=,<,API,to,define,iteration,set,>;,auto,pl_stride,=,<,API,to,define,iteration,set,>;,auto,pl_npart_cell,=,<,API,to,define,iteration,set,>;,sycl_target,-,>,queue,.,submit,([&](,sycl,::,handler,&,cgh,),{,cgh,.,parallel_for,<,>(,sycl,::,range,<1,>(,pl_iter_range,),,,[=](,sycl,::,id,<1,>,idx,),{,N,E,S,O,_,P,A,R,T,I,C,L,E,S,_,K,E,R,N,E,L,_,S,T,A,R,T,const,INT,cellx,=,N,E,S,O,_,P,A,R,T,I,C,L,E,S,_,K,E,R,N,E,L,_,C,E,L,L,;,const,INT,layerx,=,N,E,S,O,_,P,A,R,T,I,C,L,E,S,_,K,E,R,N,E,L,_,L,A,Y,E,R,;,k_P,[,cellx,][0][,layerx,],+=,0.001,*,k_V,[,cellx,][0][,layerx,];,k_P,[,cellx,][1][,layerx,],+=,0.001,*,k_V,[,cellx,][1][,layerx,];,N,E,S,O,_,P,A,R,T,I,C,L,E,S,_K,E,R,N,E,L,_,E,N,D,});,}),3,.,wait_and_throw,();,There,are,two,main,approaches,that,would,extend,this,particle,loop,interface,with,a,higher-level,user,API.,The,first,is,a,code,generation,approach,where,users,would,describe,the,particle,loop,in,a,higher,level,language,,e.g.,Python,or,Julia,,then,the,lower,level,code,would,be,generated,and,compiled.,This,code,generation,approach,would,be,motivated,by,the,original,PPMD,imple-,mentation.,The,second,approach,,which,we,applied,,is,to,define,a,high-level,particle,loop,API,directly,in,C++.,By,creating,a,particle,loop,interface,within,C++,we,eliminate,the,necessity,for,users,to,learn,a,second,programming,language.,Instead,of,creating,a,code,generation,framework,within,a,high-level,language,we,instead,implemented,an,API,within,C++,by,applying,a,templating,approach.,This,template,based,implementation,allows,a,user,to,write,a,particle,loop,directly,within,C++.,The,user,does,not,require,detailed,knowledge,of,SYCL,to,write,the,loop,and,neither,does,one,require,an,intimate,knowledge,of,template,metaprogramming.,As,the,API,is,not,dependent,on,SYCL,there,is,scope,to,replace,SYCL,as,the,computational,back-end,in,future.,Furthermore,users,are,abstracted,away,from,the,implementation,details,of,the,underlying,data,structures,that,are,accessed,by,the,particle,loop.,From,the,point,of,view,of,packaging,and,installing,NP,,by,not,using,an,additional,high-level,languages,users,are,not,burdened,with,the,task,of,installing,an,additional,programming,language,on,their,HPC,facility,of,choice.,The,trade-off,is,that,the,set,of,available,code,transformations,that,can,be,applied,purely,within,the,C++,templating,language,is,more,limited,than,those,that,can,be,applied,via,a,code,generation,framework,in,Python.,In,Listing,2,we,describe,a,particle,loop,operation,that,performs,an,identical,operation,to,the,particle,loop,defined,in,Listing,1.,This,particle,loop,is,significantly,more,approachable,than,the,original,implementation.,In,addition,to,the,improvements,for,the,user,facing,interface,there,are,additional,implementation,advantages.,The,implementation,is,free,to,adjust,how,the,particle,loop,is,executed,based,on,the,particle,distribution,presented,at,runtime.,For,example,in,simulations,where,there,is,a,high,degree,of,anisotropy,in,the,spatial,distribution,of,particle,positions,the,new,style,particle,loops,offers,a,significant,speed-up.,Listing,2:,Define,and,launch,a,new,style,particle,loop,to,apply,a,Forward,Euler,like,operation.,particle_loop,(,",advection_example,",,,particle_group,,,[=](,auto,P,,,auto,V,){,P,.,at,(0),+=,0.001,*,V,.,at,(0);,P,.,at,(1),+=,0.001,*,V,.,at,(1);,},,,Access,::,write,(,Sym,<,REAL,>(,",P,",)),,,Access,::,read,(,Sym,<,REAL,>(,",V,",)),),-,>,execute,();,Secondly,the,new,interface,demands,that,the,user,describe,exactly,how,the,computational,kernel,accesses,the,data,structure.,For,example,in,Listing,2,the,“P”,particle,data,is,accessed,with,a,“write”,access,descriptor,whilst,the,“V”,particle,data,is,accessed,with,a,“read”,access,descriptor.,The,specification,of,access,descriptors,allows,the,implementation,to,make,optimisations,by,identifying,4,which,data,is,modified,after,the,execution,of,a,loop.,In,Section,2.1.2,we,describe,how,these,access,descriptors,modify,the,behaviour,of,the,loop,for,non-particle,data.,Note,that,this,particle,loop,interface,is,independent,of,SYCL,and,the,underlying,implementation,could,be,provided,by,an,alternative,suitable,back-end,,e.g.,C++,parallel,algorithms.,In,addition,to,particle,data,,which,is,defined,on,a,per-particle,basis,,we,provide,users,with,addi-,tional,data,structures,with,which,algorithms,can,be,implemented.,These,data,structures,are,all,accessible,from,the,particle,loop,interface,and,transparently,perform,copies,between,the,compu-,tational,device,and,the,host,where,required.,The,LocalArray,data,structure,is,an,array,type,that,is,local,to,the,MPI,rank,on,which,it,is,de-,fined,,i.e.,no,MPI,communication,occurs.,The,LocalArray,type,can,be,considered,similar,to,the,std::vector,type.,The,GlobalArray,type,is,an,array,type,where,a,copy,of,the,array,is,stored,on,each,MPI,rank,and,access,to,the,array,is,collective,over,the,MPI,communicator,on,which,the,GlobalArray,is,defined.,The,CellDatConst,type,is,a,matrix,type,defined,with,fixed,size,(the,“Const”,part,of,the,name),and,data,type,for,each,cell,in,a,mesh.,A,matrix,with,a,fixed,number,of,rows,and,a,variable,number,of,columns,per,cell,can,be,created,per,cell,with,the,CellDat,data,structure.,Due,to,the,SYCL,language,restrictions,the,number,of,kernel,arguments,must,be,fixed,at,compilation,time;,the,SymVector,data,structure,applies,indirection,to,allow,a,“vector”,of,particle,properties,to,be,passed,at,runtime.,Finally,the,ParticleLoopIndex,provides,access,to,indexing,information,for,the,particle,within,the,particle,loop,kernel.,These,additional,data,structures,must,be,passed,to,a,particle,loop,with,an,access,mode,that,is,commutative.,In,practice,only,commutative,access,modes,are,defined,for,these,data,structures,and,attempting,to,pass,them,to,a,particle,loop,with,an,inappropriate,access,descriptor,will,result,in,a,compile,time,error.,We,provide,an,example,of,the,data,structure,and,the,intended,use,in,this,report,,a,table,of,the,available,access,descriptors,and,the,corresponding,behaviour,is,provided,in,the,NP,documentation.,2.1.2,LocalArray,The,LocalArray,type,is,local,to,each,MPI,rank.,No,communication,between,MPI,ranks,is,per-,formed,by,the,particle,loop.,The,LocalArray,can,be,accessed,in,particle,loops,with,“read”,and,“add”,access,modes.,In,Listing,3,we,provide,an,example,use,of,LocalArray.,Listing,3:,Example,particle,loop,with,LocalArray,access.,//,Create,a,new,LocalArray,on,the,same,SYCLTarget,as,the,//,particle,group,.,auto,local_array_add,=,std,::,make_shared,<,LocalArray,<,REAL,>,>(,particle_group,-,>,sycl_target,,,//,Compute,target,to,use,.,2,,,0,//,Number,of,elements,in,the,array,.,//,Initial,value,for,elements,.,);,//,Create,a,local,array,using,initial,values,from,a,std,::,vector,.,std,::,vector,<,REAL,>,d0,=,{1.0,,,2.0,,,3.0};,auto,local_array_read,=,std,::,make_shared,<,LocalArray,<,REAL,>,>(,5,particle_group,-,>,sycl_target,,,//,Compute,device,.,d0,//,Initial,values,and,size,definition,.,);,auto,loop,=,particle_loop,(,",l,o,cal_array_example,",,,particle_group,,,//,LA_ADD,is,the,parameter,for,the,LocalArray,with,",add,",access,and,//,LA_READ,is,the,parameter,for,the,LocalArray,with,read,-,only,access,.,[=](,auto,ID,,,auto,V,,,auto,LA_ADD,,,auto,LA_READ,){,//,Increment,the,first,component,of,LA_ADD,by,1.,LA_ADD,.,fetch_add,(0,,,1);,//,Increment,the,second,component,of,LA_ADD,with,the,entry,in,ID,[0].,LA_ADD,.,fetch_add,(1,,,ID,[0]);,//,Read,from,LA_READ,and,assign,the,values,to,the,V,particle,//,component,.,V,[0],=,LA_READ,.,at,(0);,V,[1],=,LA_READ,.,at,(1);,V,[2],=,LA_READ,.,at,(2);,},,,//,Particle,property,access,descriptors,.,Access,::,read,(,Sym,<,INT,>(,",ID,",)),,,Access,::,write,(,Sym,<,REAL,>(,",V,",)),,,//,LocalArray,access,descriptors,.,Access,::,add,(,local_array_add,),,,Access,::,read,(,local_array_read,),);,//,Execute,the,loop,.,loop,-,>,execute,();,//,Get,the,contents,of,the,local,array,in,a,std,::,vector,.,auto,vec0,=,local_array_add,-,>,get,();,2.1.3,GlobalArray,Unlike,the,LocalArray,,a,GlobalArray,is,intended,to,have,identical,values,across,MPI,ranks.,The,GlobalArray,can,be,accessed,in,“read”,and,“add”,access,modes.,When,accessed,with,“add”,access,modes,the,particle,loop,must,be,executed,collectively,across,all,MPI,ranks.,On,completion,of,the,loop,the,local,entries,of,each,GlobalArray,are,globally,combined,(Allreduce).,In,Listing,4,we,provide,an,example,use,of,GlobalArray.,6,Listing,4:,Example,particle,loop,with,GlobalArray,access.,//,Create,a,new,GlobalArray,on,the,same,SYCLTarget,as,the,particle,group,.,auto,global_array,=,std,::,make_shared,<,GlobalArray,<,REAL,>,>(,particle_group,-,>,sycl_target,,,//,Compute,target,to,use,.,1,,,0,//,Number,of,elements,in,the,array,.,//,Initial,value,for,elements,.,);,auto,loop,=,particle_loop,(,",gl,o,bal_array_example,",,,particle_group,,,[=](,auto,V,,,auto,GA,){,//,Kinetic,energy,of,this,particle,.,const,REAL,kinetic_energy,=,0.5,*,(,V,[0],*,V,[0],+,V,[1],*,V,[1]);,//,Increment,the,first,component,of,the,global,array,with,the,//,contribution,from,this,particle,.,GA,.,add,(0,,,kinetic_energy,);,},,,//,Particle,property,access,descriptor,.,Access,::,read,(,Sym,<,REAL,>(,",V,",)),,,//,GlobalArray,access,descriptor,.,Access,::,add,(,global_array,),);,//,Execute,the,loop,.,This,must,be,called,collectively,on,the,MPI,//,communicator,of,the,SYCLTarget,as,the,add,operation,is,collective,.,loop,-,>,execute,();,//,Get,the,contents,of,the,global,array,in,a,std,::,vector,.,The,first,//,element,would,be,the,kinetic,energy,of,all,particles,in,the,//,ParticleGroup,.,auto,vec0,=,global_array,-,>,get,();,2.1.4,CellDat,and,CellDatConst,The,CellDatConst,data,structure,stores,a,constant,sized,matrix,per,mesh,cell.,When,accessed,from,a,particle,loop,both,the,“read”,and,“add”,access,descriptors,expose,the,matrix,that,corre-,sponds,to,the,cell,in,which,the,particle,resides.,The,CellDat,behaves,identically,to,CellDatConst,with,the,extension,that,the,number,of,rows,in,the,matrix,is,variable,between,cells.,In,Listing,5,we,provide,an,example,use,of,CellDatConst.,Listing,5:,Example,particle,loop,with,CellDatConst,access.,//,Get,the,number,of,cells,on,this,MPI,rank,.,const,int,cell_count,=,particle_group,-,>,domain,-,>,mesh,-,>,get_cell_count,();,7,//,Create,a,3,x1,matrix,for,cell_count,cells,.,auto,g1,=,std,::,make_shared,<,CellDatConst,<,int,>,>(,particle_group,-,>,sycl_target,,,cell_count,,,3,,,1);,//,For,each,cell,get,the,current,matrix,(,uninitialised,as,we,just,//,created,the,data,structure,),and,zero,the,values,then,write,back,//,to,the,data,structure,.,for,(,int,cx,=,0;,cx,<,cell_count,;,cx,++),{,auto,cell_data,=,g1,-,>,get_cell,(,cx,);,cell_data,-,>,at,(0,,,0),=,0.0;,cell_data,-,>,at,(1,,,0),=,0.0;,cell_data,-,>,at,(2,,,0),=,0.0;,g1,-,>,set_cell,(,cx,,,cell_data,);,},auto,loop,=,particle_loop,(,",ce,ll,_d,at_,co,ns,t_,exa,mp,le,",,,particle_group,,,[=](,auto,V,,,auto,GA,){,//,Increment,the,matrix,in,each,cell,with,the,velocities,of,//,particles,in,that,cell,.,GA,.,fetch_add,(0,,,0,,,V,[0]);,GA,.,fetch_add,(1,,,0,,,V,[1]);,GA,.,fetch_add,(2,,,0,,,V,[2]);,},,,//,Particle,property,access,descriptor,.,Access,::,read,(,Sym,<,REAL,>(,",V,",)),,,//,CellDatConst,access,descriptor,.,Access,::,add,(,g1,),);,//,Execute,the,loop,.,loop,-,>,execute,();,//,After,loop,execution,the,3,x1,matrix,in,each,cell,will,contain,//,the,sum,of,the,particle,velocities,in,each,cell,.,for,(,int,cx,=,0;,cx,<,cell_count,;,cx,++),{,auto,cell_data,=,g1,-,>,get_cell,(,cx,);,//,use,cell,data,},8,2.1.5,Vectors,of,Particle,Properties,Particle,properties,are,stored,within,the,ParticleGroup,in,ParticleDat,instances.,SymVector,enables,passing,a,set,of,ParticleDat,instances,to,the,ParticleGroup,where,the,number,of,ParticleDat,instances,is,only,known,at,runtime.,The,indexing,of,the,data,structure,in,the,kernel,In,is,via,the,positional,index,of,the,ParticleDat,relative,to,the,construction,of,the,SymVector.,Listing,6,we,provide,an,example,use,of,SymVector.,Listing,6:,Example,particle,loop,with,SymVector,access.,//,These,constants,are,captured,by,value,into,the,kernel,lambda,.,const,int,ndim,=,2;,const,REAL,dt,=,0.001;,auto,loop,=,particle_loop,(,",sy,m_vector_example,",,,particle_group,,,[=](,auto,dats_vector,){,for,(,int,dimx,=0,;,dimx,<,ndim,;,dimx,++){,//,P,has,index,0,in,dats_vector,as,it,is,first,in,the,sym_vector,.,//,V,has,index,1,in,dats_vector,as,it,is,second,.,dats_vector,.,at,(0,,,dimx,),+=,dt,*,dats_vector,.,at,(1,,,dimx,);,},},,,//,We,state,that,all,ParticleDats,in,the,SymVector,are,write,access,.,Access,::,write,(,//,Helper,function,to,create,a,SymVector,.,sym_vector,<,REAL,>(,particle_group,,,//,This,argument,may,also,be,a,std,::,vector,of,Syms,.,{,Sym,<,REAL,>(,",P,",),,,Sym,<,REAL,>(,",V,",)},),),);,//,Execute,the,loop,.,loop,-,>,execute,();,2.1.6,Particle,Indexing,Helper,Functions,Some,data,structures,are,indexed,by,the,cell,and,layer,of,the,particle.,The,ParticleLoopIndex,is,a,data,structure,that,can,be,read,in,a,ParticleLoop,to,provide,this,information.,This,data,structure,is,read-only.,In,Listing,7,we,provide,an,example,use,of,ParticleLoopIndex.,Listing,7:,Example,particle,loop,with,ParticleLoopIndex,access.,auto,loop,=,particle_loop,(,9,",lo,op,_index_example,",,,particle_group,,,[=](,auto,index,){,//,Dummy,output,variable,we,store,the,indices,in,for,this,example,.,[[,maybe_unused,]],INT,tmp,;,//,The,cell,containing,the,particle,.,tmp,=,index,.,cell,;,//,The,row,(,layer,),containing,the,particle,.,tmp,=,index,.,layer,;,//,The,linear,index,of,the,particle,on,the,calling,MPI,rank,.,//,This,index,is,in,[0,,,A,-,>,get_npart_local,()).,tmp,=,index,.,ge,t_l,oc,al,_l,ine,ar,_i,nde,x,();,//,The,linear,index,of,the,particle,in,the,current,ParticleLoop,.,//,This,index,is,in,[0,,,<,size,of,ParticleLoop,iteration,set,>).,tmp,=,index,.,get_l,oop_,linea,r_in,dex,();,},,,//,Note,the,extra,{},that,creates,an,instance,of,the,type,.,Access,::,read,(,ParticleLoopIndex,{}),);,loop,-,>,execute,();,2.1.7,Particle,Products,When,modelling,atomistic,processes,it,is,necessary,to,be,able,to,create,and,destroy,particles.,In,particular,a,process,may,create,new,child,particles,from,an,existing,parent,particle.,These,child,particles,may,inherit,a,subset,of,the,parent,particle,properties,whilst,generating,new,values,for,the,remaining,properties.,The,ProductMatrix,and,DescendantProducts,types,are,data,structures,designed,as,an,intermediate,staging,area,for,particle,properties,from,which,new,particles,should,be,created.,We,consider,a,scenario,where,N,existing,particles,are,to,each,create,M,new,particles.,These,new,particles,will,ultimately,be,added,to,the,ParticleGroup,that,contains,the,original,N,particles,and,hence,each,of,the,N,M,new,particles,require,all,of,the,properties,to,be,set,to,sensible,values.,The,particle,properties,can,always,be,modified,after,the,new,particles,are,added,to,the,ParticleGroup,by,using,a,ParticleLoop,therefore,here,we,discuss,methods,to,define,particle,properties,before,the,particles,are,added.,Particles,are,created,via,the,DescendantProducts,data,structure,that,provides,space,for,each,of,the,new,particle,properties.,These,particle,properties,are,accessed,from,each,of,the,N,parent,particles,from,a,ParticleLoop,kernel.,Once,a,DescendantProducts,instance,is,populated,with,values,those,values,can,be,used,to,create,new,particles,in,the,parent,ParticleGroup,by,calling,“add,particles,local”,with,the,new,properties.,When,a,new,DescendantProducts,instance,is,created,the,number,of,output,products,M,per,exist-,ing,particle,is,specified,along,with,the,REAL,and,INT,particle,properties,that,will,be,explicitly,set,in,10,the,ParticleLoop,kernel.,This,kernel,must,also,set,the,parent,particle,of,the,new,product,particle,by,calling,“set,parent”,for,the,product,particle,to,be,included,when,add,particles,local,is,called.,The,set,parent,mechanism,provides,two,functions,,firstly,it,defines,the,product,as,a,product,that,should,be,included,when,add,particles,local,is,called.,This,set,parent,call,can,be,intentionally,omitted,to,mask,off,any,number,of,the,M,products,that,ultimately,were,not,required.,Secondly,for,particle,properties,that,are,not,defined,in,the,DescendantProducts,constructor,the,set,parent,call,defines,the,parent,particle,from,which,these,properties,should,be,copied.,The,destination,ParticleGroup,for,the,new,particle,products,has,a,set,of,particle,properties,for,each,of,the,particles,in,the,ParticleGroup.,When,the,DescendantProducts,are,are,added,via,add,particles,local,there,are,two,options.,If,the,property,is,explicitly,defined,in,the,DescendantProducts,instance,then,the,component,values,,for,all,particles,and,all,components,of,that,property,,are,copied,from,the,DescendantProducts,into,the,corresponding,ParticleDat,in,the,ParticleGroup.,If,a,property,is,defined,in,the,ParticleGroup,and,not,in,the,DescendantProducts,then,for,all,new,particle,entries,in,the,DescendantProducts,the,component,values,for,that,property,are,copied,from,the,parents,specified,in,the,DescendantProducts.,Note,that,the,decision,to,copy,property,values,from,a,parent,particle,or,from,a,DescendantProducts,entry,is,taken,property,wise,for,all,particles,for,all,components,of,the,property.,If,finer,control,is,required,,e.g.,to,inherit,particle,properties,on,a,per,particle,or,per,component,level,,then,the,user,should,specify,the,property,in,the,DescendantProducts,instance,and,populate,the,new,properties,for,all,particles,and,components,in,the,ParticleLoop,kernel,that,is,responsible,for,populating,the,entries,in,the,DescendantProducts,instance.,In,the,ParticleLoop,kernel,that,sets,the,properties,of,the,new,products,the,property,,and,com-,ponent,,are,specified,by,an,integer,index,instead,of,the,Sym,objects,used,in,host,code.,The,ordering,of,the,properties,is,defined,as,the,order,in,which,particle,properties,are,specified,for,the,DescendantProducts,specification.,This,ordering,is,contiguous,within,all,properties,of,the,same,data,type,,e.g.,if,two,INT,properties,are,specified,then,the,integer,properties,are,indexed,with,0,and,1,and,if,three,REAL,properties,are,specified,then,the,real,valued,properties,are,indexed,0,1,and,2.,Interlacing,of,INT,and,REAL,properties,in,the,specification,is,ignored,,the,properties,are,In,indexed,within,the,set,of,properties,that,have,the,same,type,in,the,order,they,are,specified.,Listing,8,we,provide,and,example,of,the,API,for,creating,new,child,particles.,Listing,8:,Creating,new,child,particles,from,parent,particles.,/*,For,this,example,we,assume,that,the,particle,group,has,the,following,REAL,and,INT,properties,with,the,specified,number,of,components,:,REAL,----,P,2,(,positions,),V,3,Q,1,INT,---,11,CELL_ID,1,(,cell,id,),ID,1,*/,//,We,create,a,DescendantProducts,with,the,following,specification,.,//,The,number,of,components,for,a,property,must,match,the,number,of,//,components,in,the,ParticleGroup,the,products,are,added,to,.,auto,product_spec,=,product_matrix_spec,(,ParticleSpec,(,ParticleProp,(,Sym,<,REAL,>(,",V,",),,,3),,,ParticleProp,(,Sym,<,REAL,>(,",Q,",),,,1),,,ParticleProp,(,Sym,<,INT,>(,",ID,",),,,1),),);,/*,Re,-,visiting,the,properties,in,the,particle,group,and,the,product,specification,:,REAL,----,P,:,Is,a,ParticleGroup,property,not,in,the,product,spec,therefore,the,values,will,be,copied,from,the,parent,particles,.,V,:,Is,defined,in,the,product,spec,and,the,values,set,in,the,DescendantProducts,will,be,used,for,the,new,particles,.,Q,:,Is,defined,in,the,product,spec,and,the,values,set,in,the,DescendantProducts,will,be,used,for,the,new,particles,.,INT,---,CELL_ID,:,Is,a,ParticleGroup,property,not,in,the,product,spec,therefore,the,values,will,be,copied,from,the,parent,particles,.,ID,:,Is,defined,in,the,product,spec,and,the,values,set,in,the,DescendantProducts,will,be,used,for,the,new,particles,.,*/,//,Create,a,DescendantProdu,cts,with,the,above,product,spec,for,at,most,2,//,products,per,parent,particle,.,const,int,n,u,m,_,p,r,o,d,u,c,t,s,_,p,e,r,_,p,a,r,t,i,c,l,e,=,2;,auto,dp,=,std,::,make_shared,<,DescendantProducts,>(,particle_group,-,>,sycl_target,,,product_spec,,,n,u,m,_,p,r,o,d,u,c,t,s,_,pe,r,_,p,a,r,t,i,c,l,e,);,//,Define,a,ParticleLoop,which,creates,the,products,from,the,parent,//,particles,.,12,auto,loop,=,particle_loop,(,particle_group,,,[=](,auto,DP,,,auto,parent_index,,,auto,V,,,auto,Q,,,auto,ID,){,for,(,int,childx,=0,;,childx,<,n,u,m,_,p,r,o,d,u,c,t,s,_,p,e,r,_,p,a,r,ti,c,l,e,;,childx,++){,//,Enable,this,product,by,calling,set_parent,DP,.,set_parent,(,parent_index,,,childx,);,//,The,V,property,was,the,first,REAL,product,we,specified,//,and,therefore,has,property,index,0,for,at_real,.,const,int,V_index,=,0;,for,(,int,dimx,=0,;,dimx,<3,;,dimx,++){,//,Copy,V,from,parent,but,negate,the,sign,.,DP,.,at_real,(,parent_index,,,childx,,,V_index,,,dimx,),=,-1.0,*,V,.,at,(,dimx,);,},//,The,Q,property,was,the,second,REAL,product,specified,//,and,hence,has,property,index,1,for,set_real,.,const,int,Q_index,=,1;,//,Simply,copy,the,parent,Q,value,in,this,kernel,.,DP,.,at_real,(,parent_index,,,childx,,,Q_index,,,0),=,Q,.,at,(0);,//,The,ID,property,is,the,first,INT,property,we,specified,//,and,hence,has,index,0,for,at_int,.,const,int,ID_index,=,0;,//,Copy,parent,ID,but,modify,it,.,DP,.,at_int,(,parent_index,,,childx,,,ID_index,,,0),=,-1,*,ID,.,at,(0),-,100,*,childx,;,},},,,Access,::,write,(,dp,),,,Access,::,read,(,ParticleLoopIndex,{}),,,Access,::,read,(,Sym,<,REAL,>(,",V,",)),,,Access,::,read,(,Sym,<,REAL,>(,",Q,",)),,,Access,::,read,(,Sym,<,INT,>(,",ID,",)),);,//,Before,a,loop,is,executed,that,accesses,a,DescendantProducts,//,data,structure,the,reset,method,must,be,called,with,the,//,number,of,particles,in,the,iteration,set,of,the,loop,.,dp,-,>,reset,(,particle_group,-,>,get_npart_local,());,//,Execute,the,loop,to,create,the,products,.,loop,-,>,execute,();,13,//,Finally,add,the,new,products,to,the,ParticleGroup,.,particle_group,-,>,add_particles_local,(,dp,);,2.1.8,ParticleSubGroup,The,NumPy,project,is,the,de-facto,library,for,array,operations,in,the,Python,ecosystem.,A,useful,feature,NumPy,offers,is,the,ability,to,create,“views”,into,existing,arrays.,A,view,into,an,array,presents,a,subset,of,the,original,array,as,a,new,array,instance,without,copying,elements.,Selection,of,this,subset,of,elements,may,be,non-trivial,and,cumbersome,to,use.,By,creating,a,view,the,process,of,applying,an,operation,to,a,subset,of,the,elements,is,simplified,for,the,user,without,creating,additional,copies,of,the,data.,In,NP,we,offer,the,ParticleSubGroup,class,that,selects,a,subset,of,the,particles,held,in,a,parent,ParticleGroup.,A,user,creates,a,sub,group,by,specifying,a,conditional,that,evaluates,to,true,for,all,particles,in,the,sub,group,and,false,for,those,that,are,not.,This,conditional,is,a,read-only,function,of,particle,data.,On,creation,of,a,sub,group,the,implementation,records,the,indices,of,particles,for,which,the,conditional,holds,,no,particle,data,is,copied,to,create,the,sub,group.,The,implementation,uses,the,access,descriptors,we,described,in,Section,2.1.1,to,determine,when,the,stored,representation,of,the,sub,group,has,been,invalidated,by,the,modification,of,particle,data,or,other,events,,such,as,moving,particles,between,cells,or,MPI,ranks.,Listing,9:,Create,a,new,ParticleSubGroup,containing,particles,with,even,particle,ids,then,apply,a,Forward,Euler,like,operation,to,only,those,particles.,//,Create,a,ParticleSubGroup,from,even,values,of,ID,.,auto,sub_group,=,particle_sub_group,(,particle_group,,,[=](,auto,ID,),{,return,(,ID,[0],%,2,==,0);,},,,Access,::,read,(,Sym,<,INT,>(,",ID,",)),);,//,Perform,a,position,update,style,kernel,on,particles,//,with,even,values,of,ID,[0].,auto,loop,=,particle_loop,(,sub_group,,,[=](,auto,V,,,auto,P,){,P,[0],+=,0.001,*,V,[0];,P,[1],+=,0.001,*,V,[1];,},,,Access,::,read,(,Sym,<,REAL,>(,",V,",)),,,Access,::,write,(,Sym,<,REAL,>(,",P,",)),);,14,loop,-,>,execute,();,A,sub,group,may,be,used,in,place,of,a,particle,group,in,numerous,ways.,Most,interestingly,a,sub,group,is,a,valid,iteration,set,for,a,particle,loop,operation.,In,Listing,9,we,demonstrate,the,definition,of,a,particle,sub,group,that,contains,particles,with,an,even,value,in,a,hypothetical,“ID”,particle,integer,property.,We,also,demonstrate,the,application,of,a,particle,loop,that,executes,over,the,particles,in,this,sub,group.,A,user,could,implement,the,example,in,Listing,9,with,a,single,particle,loop,that,combines,the,con-,ditional,that,creates,the,sub,group,and,the,Forward,Euler,kernel.,The,sub,group,is,computationally,advantageous,when,a,subset,of,a,particle,group,is,selected,,in,a,particle,sub,group,,then,this,sub,group,is,reused,for,multiple,subsequent,operations.,An,efficient,interface,exists,to,create,a,sub,group,that,contains,every,particle,in,the,parent,particle,group.,This,functionality,allows,users,to,create,generic,interfaces,that,expect,to,be,passed,sub,groups,instead,of,particle,groups.,Sub,groups,may,be,created,from,sub,groups,allowing,a,hierarchy,of,views,into,a,single,parent,particle,group,to,be,created.,Furthermore,a,user,may,specify,which,particles,are,to,be,removed,from,a,particle,group,or,alternatively,added,to,a,different,particle,group,by,passing,a,sub,group.,An,example,use,case,would,be,to,first,identify,particles,in,the,simulation,that,have,very,small,com-,putational,weight,then,combine,these,particles,into,a,single,particle,per,cell.,This,amalgamation,process,can,be,achieved,using,the,additional,data,structures,we,described,in,Sections,2.1.2-2.1.6,where,the,exact,process,will,depend,on,how,the,user,designs,the,process,to,occur.,2.2,NESO,Composite-Particle,Interaction,Interface,Our,existing,numerical,experiments,,that,contain,particles,,have,applied,simplistic,periodic,bound-,ary,conditions,in,computational,domains,where,identifying,the,particles,that,have,left,the,domain,is,computationally,simple,(i.e.,working,with,cuboids,is,easy).,The,boundary,of,a,cuboid,domain,is,exactly,representable,by,a,linear,3-D,mesh,,with,this,property,we,can,simply,compare,particle,positions,with,the,cuboid,origin,and,extents,to,determine,if,a,particle,has,crossed,a,boundary.,If,we,consider,a,more,complex,target,geometry,,for,example,a,torus,,there,is,a,discrepancy,be-,tween,the,desired,computational,domain,and,the,discrete,approximation,achieved,by,the,mesh.,In,Figure,1,we,present,a,coarse,approximation,of,a,torus,constructed,from,a,collection,of,tetrahedra.,In,this,torus,geometry,it,is,insufficient,to,determine,if,a,particle,has,crossed,the,domain,bound-,ary,by,comparing,the,particle,position,with,the,original,,i.e.,pre-meshing,,definition,of,the,torus.,With,a,more,complex,domain,it,is,unrealistic,to,compare,particle,positions,with,the,original,CAD,description.,In,practice,it,is,highly,desirable,for,a,user,to,specify,different,boundary,conditions,for,each,surface,in,the,computational,domain.,For,example,consider,a,cylinder,domain,with,particle,flow,along,the,axis,of,the,cylinder.,To,model,the,relevant,physical,system,the,user,may,specify,that,the,particles,that,cross,the,ends,of,the,domain,are,deleted,and,that,particles,that,cross,the,sides,of,the,domain,are,reflected,back,into,the,domain.,The,extensive,set,of,possible,boundary,conditions,a,user,may,apply,motivates,a,library,design,where,the,list,of,particles,that,crossed,each,boundary,in,a,move,operation,are,provided,to,the,15,Figure,1:,Wireframe,of,torus,mesh,composed,of,tetrahedra.,user,and,it,is,the,responsibility,of,the,user,to,describe,exactly,how,these,particles,are,treated.,In,Nektar++,boundary,conditions,,for,the,finite,element,method,,are,specified,on,a,per,“composite”,basis,where,a,composite,is,a,collection,of,elements,of,the,same,type.,For,example,in,the,torus,presented,in,Figure,1,the,composite,with,id,“100”,is,the,set,of,triangles,that,form,the,surface.,In,NP,we,provide,the,particle,sub,group,class,as,a,lightweight,view,into,a,particle,group,formed,from,all,particles,that,satisfy,a,condition.,The,composite,interaction,capability,in,NESO,,which,we,will,now,describe,,creates,a,map,from,composite,ids,,which,were,specified,by,the,user,,to,the,sub,group,of,particles,that,passed,through,the,composite,in,the,last,time,step.,As,particle,sub,groups,are,permissible,iteration,sets,for,particle,loops,,a,user,describes,the,physical,process,to,apply,on,each,boundary,in,terms,of,particle,loops.,In,Figure,2,we,present,a,snapshot,of,a,system,where,particles,are,transported,inside,a,torus,and,are,reflected,upon,contact,with,the,torus,wall.,The,NESO,implementation,provides,the,set,of,particles,that,intersect,each,composite,as,a,particle,sub,group,,using,this,functionality,we,implement,an,algorithm,to,apply,a,reflection,process,to,each,particle.,This,reflection,process,,and,corresponding,advection,operator,,is,an,example,of,a,process,where,a,set,of,operations,is,iteratively,repeated,at,each,time,step,until,all,particles,in,the,system,have,completed,a,time,step.,In,Algorithm,1,we,present,the,high-level,overview,of,a,time,step,with,reflective,boundary,conditions.,Algorithm,1,assumes,that,there,is,an,implementation,to,detect,and,compute,the,intersection,of,particle,trajectories,and,mesh,composites.,For,the,implementation,we,describe,in,this,report,,we,assume,that,the,mesh,is,linear,and,hence,all,the,2-D,mesh,elements,that,form,the,surface,of,the,3-D,mesh,each,exist,in,a,plane.,To,determine,if,a,line,segment,intersects,a,given,triangle,or,quadrilateral,we,first,determine,the,intersection,of,the,line,segment,and,the,plane,that,contains,the,element.,Once,that,intersection,point,is,computed,we,determine,if,the,point,lies,within,the,geometry,object,itself,by,repurposing,the,Newton,iteration,implementation,that,exists,to,bin,particles,into,mesh,elements.,16,Figure,2:,Wireframe,of,torus,mesh,composed,of,tetrahedra,containing,particles.,To,evaluate,the,Newton,iterations,required,to,determine,if,a,point,lies,within,a,surface,element,we,require,the,description,of,the,surface,element,itself.,As,is,typical,of,MPI,parallelised,FEM,simu-,lation,,the,mesh,itself,has,undergone,a,domain,decomposition,process,and,is,distributed,across,the,set,of,MPI,ranks.,We,assume,that,for,a,scientifically,interesting,simulation,it,is,impracticable,,due,to,memory,constraints,,to,store,all,surface,elements,on,all,MPI,ranks.,Secondly,we,require,an,implementation,that,correctly,accounts,for,the,scenario,where,a,particle,travels,over,an,MPI,decomposition,boundary,,i.e.,⃗ri,and,⃗r′,i,are,points,on,different,MPI,ranks,,and,in,this,one,step,the,trajectory,intersects,a,surface,element,owned,by,a,remote,MPI,rank.,To,collect,the,elements,with,which,a,particle,trajectory,potentially,intersects,,we,use,the,coarse,Cartesian,mesh,described,in,[3].,This,coarse,mesh,has,the,property,that,each,cell,is,uniquely,owned,by,an,MPI,rank,and,this,owning,rank,is,known,globally.,We,build,a,map,from,each,Cartesian,cell,to,the,2-D,surface,elements,that,intersect,the,Cartesian,cell.,By,determining,the,Cartesian,cells,intersected,by,the,line,segment,we,determine,the,candidate,2-D,surface,elements,to,inspect,for,intersections,with,the,particle,trajectory.,In,Algorithm,2,we,provide,a,high-level,description,of,the,setup,process,required,to,map,Cartesian,cells,to,2-D,surface,elements.,Secondly,,in,Algorithm,3,we,provide,an,overview,of,the,process,to,determine,if,a,particle,trajectory,intersects,a,composite.,17,Set,ti,Set,t′,Set,a,while,0,0,ParticleSubGroup(i,>,0,do,for,particle,i,←,i,←,←,a,|,|,A),∈,a,do,ti,−,∈,Set,δti,=,δt,Set,⃗r′,⃗ri,⃗ri,+,δti⃗vi,Set,⃗ri,ti,+,δti,Set,ti,Set,t′,δti,i,←,←,←,i,←,end,b,ParticleSubGroup(i,←,surface,composite),b,do,for,particle,i,a,where,the,line,segment,between,⃗r′,i,and,⃗ri,intersects,a,∈,∈,i,and,⃗ri,and,the,surface,Read,⃗ci,,the,intersection,of,the,segment,between,⃗r′,composite,Read,⃗ni,,the,normal,to,the,surface,at,⃗ci,Set,⃗vi,from,reflection,computed,with,surface,normal,⃗ni,Compute,proportion,of,time,step,completed,before,intersection,with,surface,p,=,Set,⃗ri,Set,ti,Set,t′,|⃗ci−⃗r′,i|,|⃗ri−⃗r′,i|,⃗ci,ti,+,(p,pt′,i,1)t′,i,−,←,←,i,←,end,Set,a,end,←,ParticleSubGroup(i,b,where,ti,<,δt),∈,Algorithm,1:,Overview,of,algorithm,to,apply,reflective,boundary,conditions,to,a,system,of,particles,stored,in,a,particle,group,A.,Where,for,particle,i,,⃗ri,is,the,position,,⃗r′,i,is,the,previous,position,,⃗vi,is,the,velocity,,ti,[0,,δt],records,the,progression,through,each,time,step,of,size,δt,,t′,i,records,the,time,increment,of,the,last,advection,operation.,∈,18,for,Element,e,H,do,Compute,bounding,box,b,of,e,for,Cartesian,cells,c,which,intersect,b,do,∈,Determine,owning,rank,rc,of,Cartesian,cell,c,Push,e,onto,list,of,elements,to,send,to,rc,end,end,Exchange,2-D,element,description,lists.,Such,that,the,MPI,that,owns,cell,c,holds,all,the,2-D,elements,that,intersect,c,Algorithm,2:,Overview,of,algorithm,to,build,map,from,Cartesian,cells,to,2-D,elements,that,form,composites,of,interest.,Where,H,is,a,set,of,composite,indices,of,interest.,This,algorithm,is,performed,on,each,rank,r,and,is,collective,across,all,MPI,ranks,in,the,communicator.,Reset,list,of,Cartesian,cells,C,to,collect,elements,for.,for,particle,i,do,Determine,Cartesian,cell,c,containing,⃗ri,Determine,Cartesian,cell,c′,containing,⃗r′,i,for,all,cells,ˆc,between,c′,and,c,do,Append,ˆc,to,C,end,end,for,cell,c,in,C,do,D,then,if,c,̸∈,Retrieve,all,2-D,elements,from,the,MPI,rank,that,owns,c,Append,c,to,D,end,end,for,particle,i,do,Determine,Cartesian,cell,c,containing,⃗ri,Determine,Cartesian,cell,c′,containing,⃗r′,i,Set,d,Initialise,intersection,point,⃗ci,for,all,cells,ˆc,between,c′,and,c,do,←,inf,(maximum,floating,point,value,in,practice),for,elements,e,ˆc,do,∈,Compute,intersection,point,⃗ce,of,the,line,segment,between,⃗r′,plane,containing,e.,if,⃗ce,exists,then,e,then,if,⃗ce,i,and,⃗ri,and,the,⃗r′,i|,⃗ce,←,|,∈,Set,de,−,if,de,<,d,then,de,Set,d,⃗ce,Set,⃗ci,←,←,end,end,end,end,end,end,19,Algorithm,3:,Overview,of,algorithm,to,detect,intersection,between,particle,trajectories,and,mesh,composites.,Where,for,each,particle,i:,⃗ri,is,the,current,particle,position,,⃗r′,i,is,the,previous,particle,position.,D,is,a,list,of,Cartesian,cells,that,have,been,previously,inspected,and,for,which,this,rank,holds,copies,of,all,the,element,information.,The,intersection,detection,algorithm,described,in,Algorithm,3,is,designed,to,be,robust,for,the,case,where,a,particle,travels,through,multiple,surface,elements,in,a,single,step,,e.g.,by,passing,through,a,corner.,The,process,is,collective,on,the,MPI,communicator,as,at,an,intermediate,stage,the,descriptions,of,remotely,owned,elements,are,collected.,These,elements,are,cached,for,later,use,to,avoid,repeated,communication,of,element,information.,Furthermore,the,communication,procedure,begins,with,a,global,vote,to,determine,if,any,further,communication,must,occur.,It,is,expected,that,in,a,relatively,small,number,of,time,steps,each,rank,will,contain,all,required,2-D,elements,in,the,cache,and,little,further,communication,will,be,required.,2.3,Grantee,Work,Part,of,the,collaboration,with,the,University,of,Warwick,and,University,of,York,investigates,the,development,of,a,Domain,Specific,Language,(DSL),for,particle,based,operations.,The,reports,[4,,5],propose,a,DSL,for,describing,particle,operations,and,begin,to,investigate,the,performance,of,the,implementation,in,select,proxy,applications.,The,proposed,DSL,formulates,the,particle,loop,abstraction,within,the,existing,OP2,[6],implemen-,tation.,The,extension,which,adds,the,particle,loop,abstraction,to,the,OP-DSL,ecosystem,is,devel-,oped,under,the,name,“OP-PIC”.,Within,the,OP-DSL,ecosystem,,parallel,loops,are,executed,over,iteration,sets,described,by,sets,of,data.,Furthermore,,maps,may,be,described,between,different,sets.,For,example,there,may,exist,a,set,which,indexes,mesh,elements,and,a,set,which,indexes,the,mesh,vertices.,A,map,can,then,be,constructed,from,mesh,element,to,vertices,which,touch,the,mesh,cell.,Data,can,be,assigned,to,each,element,of,a,set,in,a,so,called,“dat”.,In,the,OP-DSL,ecosystem,a,parallel,loop,is,executed,over,a,set,,for,example,the,set,of,mesh,elements.,For,each,element,in,the,iteration,set,a,computational,kernel,is,executed,which,may,access,the,associated,data,and,maps,for,the,element,in,the,set.,These,kernels,must,be,suitable,for,parallel,execution.,In,the,OP-PIC,extension,a,set,of,particles,may,be,defined,and,data,is,assigned,to,each,particle.,These,particle,sets,are,valid,iteration,sets,for,an,OP-PIC,parallel,loop.,With,these,primitive,types,and,loops,the,particle,loop,is,now,implemented,within,the,OP-DSL,ecosystem.,By,defining,maps,from,sets,of,particles,to,mesh,elements,the,OP-PIC,framework,allows,users,to,describe,how,the,data,movement,must,occur,to,describe,the,conversion,from,particle,based,representations,to,mesh,based,representations.,For,example,a,charge,deposition,process,onto,a,DG0,function,space,,where,each,cell,contains,a,constant,valued,function,,can,be,performed,by,creating,the,map,mesh,cell,to,particles,contained,within,the,cell.,Each,cell,is,then,assigned,a,“dat”,which,holds,the,single,DOF,per,cell.,The,charge,deposition,loop,then,loops,over,each,cell,and,traverses,the,map,from,cells,to,particles,where,then,the,DOF,for,the,cell,is,incremented,with,the,contribution,for,each,particle,within,the,cell.,For,each,described,particle,loop,the,proposed,implementation,would,use,the,OP-PIC,framework,to,generate,the,source,code,which,performs,the,looping,operation,requested,by,the,user.,OP-PIC,may,then,generate,different,implementations,for,different,architectures,as,required.,The,OP-PIC,framework,must,provide,particle,movement,implementation,to,move,particles,over,the,decom-,posed,mesh.,20,The,particle,movement,operation,is,a,non-trivial,process,to,efficiently,and,robustly,implement.,To,move,particles,between,nearby,MPI,ranks,the,OP-PIC,framework,implements,a,“multi-hop”,process,to,identify,the,new,MPI,rank,which,owns,a,particle,when,it,leaves,the,portion,of,mesh,owned,by,the,previous,MPI,rank.,This,multi-hop,process,is,extended,with,a,“direct-hop”,process,which,,like,NESO-Particles,,adds,an,additional,Cartesian,mesh,over,the,unstructured,mesh,to,facilitate,the,movement,of,particles,which,take,large,steps,in,space,per,time,step.,The,reports,present,timing,results,from,the,OP-PIC,implementation,where,the,authors,compare,a,charge,deposition,process,on,multiple,CPU,and,GPU,architectures.,This,charge,deposition,process,is,an,interesting,test,case,as,an,efficient,implementation,must,overcome,a,parallel,write,contention,when,many,particles,contribute,to,a,single,mesh,DOF.,The,optimal,way,to,perform,this,reduction,is,architecture,dependent.,Furthermore,the,report,presents,weak,scaling,results,from,CPU,and,GPU,platforms,to,demonstrate,the,weak,scaling,performance,of,the,particle,movement,process,implemented,in,OP-PIC.,2.4,Strong,Scaling,Experiments,As,part,of,the,software,development,process,for,HPC,it,is,important,to,continuously,evaluate,and,analyse,the,performance,of,the,implementation.,In,previous,strong,scaling,studies,[7],we,established,that,the,strong,scaling,limit,,i.e.,the,smallest,problem,size,for,which,good,parallel,efficiency,is,maintained,,was,approximately,105,particles,per,CPU,core,for,a,two-stream,instability,example.,To,ensure,we,maintain,at,least,this,level,of,performance,we,constructed,a,micro-benchmark,which,transports,a,large,number,of,particles,over,a,square,domain,constructed,as,a,Cartesian,mesh.,This,Cartesian,mesh,has,the,feature,that,mapping,particle,positions,is,relatively,cheap,and,hence,most,of,the,runtime,of,the,benchmark,is,in,the,particle,bookkeeping,routines.,The,bookkeep-,ing,routines,are,common,to,all,simulations,which,use,NP,and,hence,the,performance,of,these,implementations,is,important.,In,Figure,3,we,present,the,results,of,a,strong,scaling,experiment,on,the,ARCHER2,HPC,facil-,ity.,These,results,use,AdaptiveCPP,(OpenMP,library,back-end,,GCC,11.2.0,,cray-mpich,8.1.23),as,the,SYCL,implementation.,Although,this,SYCL,implementation,is,OpenMP,based,,we,use,a,single,OpenMP,thread,per,CPU,core.,With,this,configuration,we,launch,4096,MPI,ranks,when,run-,ning,the,experiments,on,32,nodes.,The,size,of,the,computational,problem,was,chosen,to,reach,approximately,105,particles,per,core,when,16,nodes,are,used.,ARCHER2,is,a,HPE,Cray,EX,sys-,tem,where,computational,nodes,are,comprised,of,two,EPYC,7742,64-core,2.25GHz,processors.,These,results,demonstrate,good,parallel,efficiency,(90%+),with,105,particles,per,core.,2.5,Recombination,A,natural,next,addition,to,the,capabilities,of,the,NESO,software,is,to,add,the,leading,opposing,process,to,electron-impact,ionisation,,namely,radiative,recombination[8]:,e,+,H,+,H,+,hν,,→,21,(1),108,Figure,3:,Strong,scaling,advection,experiment,on,ARCHER2.,particles,are,transported,over,a,Cartesian,mesh,with,512,cells,in,each,dimension.,In,the,bottom,,108,particles,are,transported,over,a,mesh,with,256,cells,in,each,dimension.,Note,blue,,line,1,that,both,experiments,feature,the,same,number,of,particles,per,mesh,cell.,In-figure,numbers,are,the,parallel,efficiency,relative,to,a,single,node.,Top,figure,axis,lists,the,particles,per,core,for,the,experiment,with,4,In,the,top,,orange,,line,4,108,particles.,×,×,×,where,strictly,speaking,the,resulting,Hydrogen,atom,is,a,function,of,the,primary,quantum,number,n.,However,,in,this,proof-of-principle,work,we,take,the,assumption,that,n,=,1,and,that,it,is,safe,to,ignore,hν.,The,derivation,of,the,rate,coefficients,for,Maxwellian,electron,and,proton,distributions,follows,natu-,rally,by,applying,the,detailed,balance,principle,[8]:,i.e.,the,rate,coefficient,is,such,that,local,thermal,equilibrium,is,maintained,in,the,presence,of,ionisation.,The,expression,for,the,recombination,rate,,αrec,,is,αrec(nl),=,Anl,n,β3/2,b,βn,+,ξnl,(βn),(2),where,βn,=,In/T,,,In,=,Ry/n2,,ξ(n=0,l=s),=,0.25,,is,the,Bohr,radius,,Ry,=,13.6,eV,is,the,Rydberg,constant.,In,this,report,,we,continue,to,use,the,implementation,of,the,electron-impact,ionisation,process,as,detailed,in,the,last,milestone,M4c.3.,It,was,the,first,reaction,process,between,the,fluid-plasma,and,the,particle-neutrals,and,we,refer,the,reader,back,to,the,previous,report,[9],for,further,information.,A,discussion,of,the,implementation,details,behind,the,newly,developed,the,interaction,capability,22,12481632NumberofNodes10−1100Timeperstep(s)100.098.695.692.283.368.1100.0100.197.796.593.387.01.0·108particles4.0·108particles3.1·1061.6·1067.8·1053.9·1052.0·1059.8·104Particlespercore,follows,in,the,next,section.,2.5.1,The,recombination,ParticleLoop,Developments,made,to,the,NESO-Particles,performance,portable,particle,library,,as,described,above,,have,brought,easier,and,more,robust,methods,to,develop,particle,algorithms,on,accelerator,style,devices,like,GPUs.,In,this,subsection,we,show,one,code,snippet,,used,in,NESO,,to,elucidate,both,the,algorithm,behind,recombination,and,its,implementation.,Referring,to,Listing,10,,lines,2-7,are,dedicated,to,calculating,quantities,needed,in,the,loop,or,deliberately,making,copy,assignments,of,other,variables.,These,are,all,constant,values,,which,may,then,be,used,inside,the,lambda,function.,Line,11,is,where,the,ParticleLoop,begins,and,its,first,three,arguments,are:,a,string,to,name,the,loop,for,profiling,purposes,on,line,12;,the,ParticleGroup,object,that,the,Loop,will,traverse,and,operate,upon,on,line,13;,and,the,lambda,function,itself,beginning,on,line,14,,which,encapsulates,the,logic,to,applied,to,the,Particles,in,the,ParticleGroup.,The,final,arguments,on,lines,24-27,,inclusive,,are,the,access,descriptors,to,the,ParticleDat,instances,that,correspond,to,the,selfsame,ordered,arguments,to,the,lambda,function,on,line,14.,The,access,descriptors,define,the,type,of,the,argument,and,whether,or,not,it,will,be,mutated.,The,logic,of,the,ParticleLoop,in,lines,15-21,simply,calculates,the,reaction,rates,and,the,weight,of,the,newly,created,recombined,neutral,particle,and,the,amount,of,plasma,density,that,is,duly,lost,in,the,process.,Finally,on,line,30,,the,ParticleLoop,is,executed,and,the,ParticleGroup,is,operated,upon,according,the,logic,defined,in,the,lambda,function.,After,execution,,the,ParticleDat,instances,will,be,filled,with,the,appropriately,updated,values,required,for,logic,elsewhere,in,the,program.,23,2.5.2,Turbulence,simulations,with,ionisation,and,recombination,physics,In,this,section,we,present,simulations,of,3-D,fluid,turbulence,using,higher,order,finite,elements,combined,with,GPU,capable,neutral,particle,physics.,The,plasma,turbulence,model,that,we,use,is,the,Hasegawa-Wakatani,model,with,an,additional,source,term,for,plasma,density,,Sn,,arising,from,the,evolution,of,the,neutral,particles.,These,equations,for,the,plasma,density,n,and,potential,ϕ,are,∂n,∂t,=,n,,ϕ,{,},−,κ,∂ϕ,∂y,+,α,2,∥(n,∇,−,ϕ),+,Sn,,∂ζ,∂t,=,ζ,,ϕ,{,},+,α,2,∥(n,∇,−,ϕ),,(3),(4),∇,2,⊥ϕ,is,the,vorticity,,where,ζ,=,x,,y,represents,the,Poisson,bracket,operator,in,the,perpendicular,directions,x,and,y,,and,the,parallel,direction,aligns,with,z.,α,=,T0,scales,the,parallel,dynamics,in,the,model,,which,itself,is,determined,by,the,electron,response.,Here,ωci,=,ZiqiB/mi,is,the,ion,cyclotron,frequency,and,η,is,the,well,known,Spitzer,electron,resistivity,[10].,κ,=,ρs0,determines,λq,n0eηωci,},{,√,miTe,eB,is,the,hybrid,Larmor,radius,and,λq,=,1,n0,∂N,∂x,,,the,radial,density,scale,length,,where,ρs0,=,which,is,the,heat,flux,width.,Note,that,this,set,of,equations,adds,new,physics,to,previous,3-D,implementations,of,the,Hasegawa-,Wakatani,equations,in,that,previously,there,was,no,parallel,diffusion,term,,just,the,term,α(n,ϕ).,With,parallel,diffusion,switched,on,,these,equations,embody,one,more,logical,step,towards,a,full,plasma,model,since,it,enables,us,to,study,the,effects,of,parallel-diffusion,on,the,code’s,stability.,It,was,found,that,the,parallel-diffusion,term,has,a,stabilising,effect,on,the,numerics,,as,expected.,−,Listing,2.5.2,shows,the,physics-,and,meta-parameters,of,the,proof-of-principle,simulation,,results,are,plotted,in,Figures,4,and,5,for,the,case,without,recombination,and,with,recombination,respec-,tively.,Recombination,is,switched,off,by,setting,num,recombination,particles,per,cell,per,step,=,0,since,the,physical,rates,are,determined,from,the,plasma,state,and,not,configured,by,the,user.,Referring,to,the,upper-right,panel,of,Figs.,4,and,5,,both,showing,the,snapshot,in,time,of,the,electron,density,at,the,late,non-linear,stage,of,the,simulation,,it,can,be,seen,that,the,structures,are,more,diffuse,without,recombination,physics.,One,lesson,to,take,away,from,this,is,to,ensure,that,the,spatial,distribution,of,the,electron,density,remains,smooth,through,choosing,a,suitably,large,number,of,neutral,particles,considering,that,1/√N,particle,noise,can,sometimes,cause,large,oscillations,in,the,higher,order,basis,functions.,A,subject,of,future,work,is,on,the,topic,of,efficient,use,of,particles,such,as,importance,sampling,,splitting,and,roulette,and,low,discrepancy,numbers,to,ensure,that,this,problem,is,mitigated,on,multiple,fronts.,These,techniques,will,allow,us,to,retain,the,good,signal,to,noise,ratios,and,alleviate,some,of,the,associated,computational,cost.,24,4,1,-,e,1,4,3,2,0,.,1,s,i,e,t,a,r,n,o,i,t,a,s,i,n,o,i,/,/,;,e,t,a,r,_,n,o,i,t,a,n,i,b,m,o,c,e,r,_,m,/,/,;,),V,e,_,e,r,u,t,a,r,e,p,m,e,t,(,e,t,a,r,_,n,o,i,t,a,n,i,b,m,o,c,e,r,=,e,t,a,r,;,e,m,u,l,o,v,_,o,d,u,e,s,p,_,e,l,c,i,t,r,a,p,_,k,*,I,S,_,n,*,I,S,_,n,*,I,S,_,t,d,_,k,*,e,t,a,r,=,t,h,g,i,e,w,L,A,E,R,L,A,E,R,{,),Y,T,I,S,N,E,D,_,E,C,R,U,O,S,o,t,u,a,,,T,H,G,I,E,W,_,L,A,N,O,I,T,A,T,U,P,M,O,C,o,t,u,a,,,D,I,_,E,L,C,I,T,R,A,P,o,t,u,a,,,Y,T,I,S,N,E,D,_,N,O,R,T,C,E,L,E,o,t,u,a,(,],=,[,;,),0,(,t,a,.,Y,T,I,S,N,E,D,_,N,O,R,T,C,E,L,E,=,I,S,_,n,o,t,u,a,t,s,n,o,c,{,),0,<,),0,(,t,a,.,D,I,_,E,L,C,I,T,R,A,P,(,f,i,s,s,a,m,s,e,s,o,l,a,m,s,a,l,p,/,/,;,t,d,_,k,/,e,l,a,c,s,_,n,_,k,*,t,h,g,i,e,w,=,-,),0,(,t,a,.,Y,T,I,S,N,E,D,_,E,C,R,U,O,S,t,h,g,i,e,w,l,a,r,t,u,e,n,/,/,;,t,h,g,i,e,w,=,),0,(,t,a,.,T,H,G,I,E,W,_,L,A,N,O,I,T,A,T,U,P,M,O,C,e,v,i,t,a,g,e,n,r,e,g,n,o,l,o,n,/,/,;,1,-,=,*,),0,(,t,a,.,D,I,_,E,L,C,I,T,R,A,P,f,o,s,e,u,l,a,v,n,e,v,e,h,t,i,w,s,e,l,c,i,t,r,a,p,n,o,l,e,n,r,e,k,e,l,y,t,s,e,t,a,d,p,u,n,o,i,t,i,s,o,p,a,m,r,o,f,r,e,P,.,],0,[,D,I,/,/,/,/,,,",p,o,o,l,_,e,l,c,i,t,r,a,p,_,n,o,i,t,a,n,i,b,m,o,c,e,r,",(,p,o,o,l,_,e,l,c,i,t,r,a,p,=,p,o,o,l,o,t,u,a,,,p,u,o,r,g,_,e,l,c,i,t,r,a,p,_,m,,,),),",T,H,G,I,E,W,_,L,A,N,O,I,T,A,T,U,P,M,O,C,",(,>,L,A,E,R,<,m,y,S,(,e,t,i,r,w,:,:,s,s,e,c,c,A,),),",Y,T,I,S,N,E,D,_,E,C,R,U,O,S,",(,>,L,A,E,R,<,m,y,S,(,e,t,i,r,w,:,:,s,s,e,c,c,A,,,),),",Y,T,I,S,N,E,D,_,N,O,R,T,C,E,L,E,",(,>,L,A,E,R,<,m,y,S,(,d,a,e,r,:,:,s,s,e,c,c,A,,,),),",D,I,_,E,L,C,I,T,R,A,P,",(,>,T,N,I,<,m,y,S,(,e,t,i,r,w,:,:,s,s,e,c,c,A,},,,},;,),(,e,t,u,c,e,x,e,>,-,p,o,o,l,;,),;,n,r,u,t,e,r,},{,),s,e,l,c,i,t,r,a,p,_,b,m,o,c,e,r,_,d,e,d,d,a,_,m,u,n,t,n,i,t,s,n,o,c,,,t,d,e,l,b,u,o,d,t,s,n,o,c,(,s,d,l,e,i,f,_,e,t,a,u,l,a,v,e,_,t,s,o,p,_,n,o,i,t,a,n,i,b,m,o,c,e,r,d,i,o,v,e,n,i,l,n,i,.,p,o,o,L,e,l,c,i,t,r,a,P,n,o,i,t,a,n,b,m,o,c,e,r,i,e,h,T,:,0,1,g,n,i,t,s,L,i,;,I,S,_,o,t,_,t,_,m,*,t,d,=,I,S,_,t,d,_,k,e,l,b,u,o,d,;,t,d,=,t,d,_,k,e,l,b,u,o,d,t,s,n,o,c,t,s,n,o,c,t,a,D,e,l,c,i,t,r,a,P,a,n,o,y,t,i,t,n,a,u,q,d,i,u,l,f,a,e,b,d,l,u,o,w,s,i,h,t,l,a,r,e,n,e,g,n,i,4,1,-,e,1,4,3,2,0,.,1,s,i,e,t,a,r,n,o,i,t,a,s,i,n,o,i,/,/,!,s,s,e,u,g,a,s,i,/,/,;,V,e,T,_,m,=,V,e,_,e,r,u,t,a,r,e,p,m,e,t,e,l,b,u,o,d,s,i,h,t,/,/,;,9,1,-,e,1,=,e,t,a,r,e,l,b,u,o,d,t,s,n,o,c,/,/,;,s,e,l,c,i,t,r,a,p,_,b,m,o,c,e,r,_,d,e,d,d,a,_,m,u,n,/,e,m,u,l,o,v,_,n,o,i,g,e,r,_,e,l,c,i,t,r,a,p,_,m,=,e,m,u,l,o,v,_,o,d,u,e,s,p,_,e,l,c,i,t,r,a,p,_,k,e,l,b,u,o,d,;,I,S,_,o,t,_,n,_,m,/,1,=,e,l,a,c,s,_,n,_,k,e,l,b,u,o,d,t,s,n,o,c,t,s,n,o,c,t,s,n,o,c,1,2,3,4,5,6,7,8,9,0,1,1,1,2,1,3,1,4,1,5,1,6,1,7,1,25,8,1,9,1,0,2,1,2,2,2,3,2,4,2,5,2,6,2,7,2,8,2,9,2,0,3,1,3,2,3,3,3,p,a,r,t,i,c,l,e,n,u,m,w,r,i,t,e,p,a,r,t,i,c,l,e,s,t,e,p,s,I,O,C,h,e,c,k,S,t,e,p,s,P,a,r,t,i,c,e,l,d,a,t,a,w,r,i,t,t,e,n,e,v,e,r,y,n,t,h,s,t,e,p,n,t,n,t,o,t,o,b,g,S,I,S,I,S,I,T,e,e,V,p,a,r,t,i,c,l,e,s,o,u,r,c,e,w,i,d,t,h,p,a,r,t,i,c,l,e,d,r,i,f,t,v,e,l,o,c,i,t,y,p,a,r,t,i,c,l,e,t,h,e,r,m,a,l,v,e,l,o,c,i,t,y,p,a,r,t,i,c,l,e,p,o,s,i,t,i,o,n,s,e,e,d,p,a,r,t,i,c,l,e,n,u,m,b,e,r,d,e,n,s,i,t,y,g,r,o,w,t,h,r,a,t,e,s,r,e,c,o,r,d,i,n,g,s,t,e,p,n,u,m,r,e,c,o,m,b,i,n,a,t,i,o,n,p,a,r,t,i,c,l,e,s,p,e,r,c,e,l,l,p,e,r,s,t,e,p,1,0,0,.,2,2,.,0,1,.,0,1,1,0,1,8,1,0,1,6,1,0,0,1,1,2,×,×,1,0,1,7,1,0,−,4,.,.,.,f,o,r,t,h,e,r,e,c,o,m,b,n,a,t,i,o,n,i,p,r,o,c,e,s,s,i,A,d,a,g,n,o,s,t,i,c,s,t,i,m,e,s,t,e,p,f,o,r,e,n,e,r,g,y,a,n,d,e,n,s,t,r,o,p,h,y,N,u,m,b,e,r,d,e,n,s,i,t,y,n,o,r,m,a,l,i,i,s,n,g,f,a,c,t,o,r,i,n,[,m,−,3,],i,T,m,e,n,o,r,m,a,l,i,s,a,t,i,o,n,f,a,c,t,o,r,i,n,[,s,],A,s,s,u,m,e,d,b,a,c,k,g,r,o,u,n,d,d,e,n,s,i,t,y,i,n,S,I,[,m,−,3,],T,e,m,p,e,r,a,t,u,r,e,i,n,e,V,u,s,e,d,t,o,c,o,m,p,u,t,e,i,i,o,n,s,a,t,i,o,n,r,a,t,e,T,h,e,s,p,a,t,i,a,l,s,o,u,r,c,e,w,d,t,h,i,i,n,l,i,s,m,u,a,t,i,o,n,u,n,i,t,s,f,o,r,t,h,e,i,n,i,t,i,a,l,p,a,r,t,i,c,e,l,s,e,t,T,h,e,d,r,i,f,t,s,p,e,e,d,i,n,l,i,s,m,u,a,t,i,o,n,u,n,i,t,s,f,o,r,t,h,e,i,n,i,t,i,a,l,p,a,r,t,i,c,e,l,s,e,t,S,e,e,d,f,o,r,r,a,n,d,o,m,n,u,m,b,e,r,g,e,n,e,r,a,t,o,r,T,h,e,i,n,i,t,i,a,l,b,a,c,k,g,r,o,u,n,d,n,u,m,b,e,r,d,e,n,s,i,t,y,o,f,t,h,e,n,e,u,t,r,a,s,l,T,h,e,t,h,e,r,m,a,l,s,p,e,e,d,i,i,n,s,m,u,a,t,i,o,n,l,u,n,i,t,s,f,o,r,t,h,e,i,n,i,t,i,a,l,p,a,r,t,i,c,e,l,s,e,t,d,2,2,B,x,y,s,H,W,H,W,H,W,H,W,k,a,p,p,a,a,l,p,h,a,n,u,e,i,o,m,e,g,a,c,e,I,O,I,O,I,n,f,o,S,t,e,p,s,C,h,e,c,k,S,t,e,p,s,T,F,i,n,a,l,N,u,m,S,t,e,p,s,T,i,m,e,S,t,e,p,V,a,r,i,a,b,e,l,n,u,m,p,a,r,t,i,c,l,e,s,t,o,t,a,l,n,u,m,p,a,r,t,i,c,l,e,s,p,e,r,c,e,l,l,i,n,u,m,p,a,r,t,i,c,l,e,s,t,e,p,s,p,e,r,f,l,u,i,d,s,t,e,p,6,4,0,0,0,V,a,l,u,e,0,.,0,0,0,6,2,5,T,o,t,a,l,n,u,m,b,e,r,o,f,i,t,e,r,a,t,i,o,n,s,D,e,s,c,r,i,p,t,i,o,n,i,T,m,e,p,e,r,i,t,e,r,a,t,i,o,n,i,i,n,s,m,u,a,t,i,o,n,l,u,n,i,t,s,1,0,0,0,0,0,T,o,t,a,l,n,u,m,b,e,r,o,f,p,a,r,t,i,c,e,s,l,a,t,i,n,i,t,i,a,l,i,s,a,t,i,o,n,1,−,1,0,.,5,1,.,0,0,.,1,3,.,5,0,.,1,0,.,0,1,.,0,N,u,m,S,t,e,p,s,/,1,6,0,N,u,m,S,t,e,p,s,/,1,6,0,0,×,N,u,m,S,t,e,p,s,T,i,m,e,S,t,e,p,i,F,n,a,l,t,i,m,e,i,i,n,s,m,u,a,t,i,o,n,l,u,n,i,t,s,H,W,a,p,h,a,l,p,a,r,a,m,e,t,e,r,H,W,a,p,h,a,l,p,a,r,a,m,e,t,e,r,i,T,m,e,p,e,r,i,n,f,o,d,u,m,p,C,o,e,f,f,f,o,r,l,H,e,m,h,o,l,t,z,s,o,v,e,l,M,a,g,n,e,t,i,c,fi,e,d,l,s,t,r,e,n,g,t,h,i,T,m,e,f,o,r,c,h,e,c,k,l,fi,e,d,u,m,p,H,W,p,a,r,a,m,e,t,e,r,S,c,a,l,i,n,g,f,a,c,t,o,r,f,o,r,I,C,s,l,e,e,c,t,r,o,n,l,c,y,c,o,t,r,o,n,f,r,e,q,u,e,n,c,y,N,u,m,b,e,r,o,f,n,e,u,t,r,a,l,p,a,r,t,i,c,e,s,l,p,e,r,c,e,l,l,N,u,m,b,e,r,o,f,s,u,b,s,t,e,p,s,o,f,n,e,u,t,r,a,l,l,a,g,o,r,i,t,h,m,p,e,r,fl,u,d,i,a,d,v,a,n,26,c,e,Figure,4:,Evolution,of,the,electron,density,evolving,according,to,the,3-D,Hasegawa-Wakatani,equations,in,the,presence,of,ionisation,without,recombination.,The,four,quadrants,depict,four,stages,in,the,simulation:,lower-left,,early,linear;,top-left,,late,linear;,lower-right,,early,non-linear;,top-right,,late,non-linear.,27,Figure,5:,As,in,4,except,with,the,addition,of,recombination,physics.,Note,that,recombination,alters,the,non-linear,evolution,of,the,simulation.,28,3,Summary,In,this,report,we,describe,further,developments,made,to,the,NESO,framework,enabling,the,parallel,dynamics,in,a,3-D,Hasegawa-Wakatani,system,coupled,to,neutral,particles,involving,ionisation,and,recombination,physics.,Significant,effort,has,gone,into,augmenting,the,capability,of,the,NESO-Particles,framework,so,that,the,user,may,write,performance,portable,GPU-capable,particle,kernels,completely,without,knowledge,of,SYCL,,the,performance,portability,framework,used,within,NESO-Particles,allowing,it,to,run,on,multiple,architectures,in,a,vendor,agnostic,way.,Importantly,,this,abstracts,SYCL,from,NESO,such,that,other,performance,portable,frameworks,could,be,used,to,enable,execution,of,particle,kernels,on,GPUs,without,any,breaking,changes,to,NESO,itself.,This,is,foundational,to,robust,software,that,respects,the,principle,of,separation,of,concerns.,Deploying,the,latest,changes,made,in,NESO-Particles,to,NESO,in,the,form,of,new,recombination,physics,,we,have,been,able,to,observe,fundamental,changes,to,the,nonlinear,behaviour,of,the,turbulent,plasma,state.,This,capability,is,a,prerequisite,to,accurate,modelling,of,the,exhaust,region,in,tokamaks,and,hence,their,design.,Acknowledgement,The,support,of,the,UK,Meteorological,Office,and,Strategic,Priorities,Fund,is,acknowledged.,This,work,used,the,ARCHER2,UK,National,Supercomputing,Service,(https://www.archer2.ac.uk).,References,[1],William,Robert,Saunders,,James,Grant,,and,Eike,Hermann,M,¨uller.,A,domain,specific,lan-,guage,for,performance,portable,molecular,dynamics,algorithms.,Computer,Physics,Commu-,nications,,224:119–135,,2018.,[2],W.,Saunders,,J.,Cook,,and,W.,Arter.,Technical,Report,CD/EXCALIBUR-FMS/0061-M4.2,,UKAEA,,12,rities.,https://github.com/ExCALIBUR-NEPTUNE/Documents/blob/main/reports/ukaea_,reports/CD-EXCALIBUR-FMS0061-M4.2.pdf.,2-D,Model,of,Neutral,Gas,and,Impu-,2021.,[3],W.,Saunders,,J.,Cook,,and,W.,Arter.,High-dimensional,Models,Complemen-,tary,Actions,2.,Technical,Report,CD/EXCALIBUR-FMS/0062-M4.3,,UKAEA,,3,2022.,https://github.com/ExCALIBUR-NEPTUNE/Documents/blob/main/reports/ukaea_,reports/CD-EXCALIBUR-FMS0062-M4.3.pdf.,[4],S.,Wright,,E.,Higgins,,G.,Mudalige,,Z.,Lantra,,B.F.,McMillan,,and,T.,Goffrey.,Proposal,for,a,Particle,DSL.,Technical,Report,2057699-TN-04-2,,UKAEA,Project,Neptune,,2023.,https://,github.com/ExCALIBUR-NEPTUNE/Documents/blob/main/reports/2057699/TN-04-2.pdf.,29,[5],S.,Wright,,E.,Higgins,,C.,Ridgers,,G.,Mudalige,,and,Z.,Lantra.,DSL,Mini-application,Per-,formance,Report,Revision,1.0.,Technical,Report,2067270-TN-05-1,,UKAEA,Project,Nep-,tune,,2024.,https://github.com/ExCALIBUR-NEPTUNE/Documents/blob/main/reports/,2067270/TN-05-1.pdf.,[6],G.R.,Mudalige,,M.B.,Giles,,I.,Reguly,,C.,Bertolli,,and,P.H.J,Kelly.,Op2:,An,active,library,framework,for,solving,unstructured,mesh-based,applications,on,multi-core,and,many-core,architectures.,In,2012,Innovative,Parallel,Computing,(InPar),,pages,1–12,,2012.,[7],J.,Cook,,W.,Saunders,,and,W.,Arter.,1-D,and,2-D,particle,models.,Technical,Report,CD/EXCALIBUR-FMS/0070,,UKAEA,Project,Neptune,,2022.,[8],R.,K.,Janev,,Ulrich,Samm,,and,Detlev,Reiter.,Collision,processes,in,low-temperature,hydro-,gen,plasmas.,2003.,[9],J.,Cook,,W.,Saunders,,O.,Parry,,M.,Barton,,and,W.,Arter.,3-D,integrated,particle,and,contin-,uum,model.,Technical,Report,CD/EXCALIBUR-FMS/0073,,UKAEA,Project,Neptune,,2023.,[10],Lyman,Spitzer,and,Richard,H,¨arm.,Transport,Phenomena,in,a,Completely,Ionized,Gas.,Phys-,ical,Review,,89(5):977–981,,March,1953.,30

:pdfembed:`src:_static/CD-EXCALIBUR-FMS0082-M4c.4_ExcaliburFusionModellingSystemProgressTowardsImplementation3DTurbulenceCodeTight.pdf, height:1600, width:1100, align:middle`