CD-EXCALIBUR-FMS0044-M2.5.2_SelectionModelOrderReduction
========================================================

.. meta::
   :description: technical note

   :keywords: ExCALIBUR,Selection,of,techniques,for,Model,Order,Reduction,M2.5.2,The,report,describes,work,for,ExCALIBUR,project,NEPTUNE,at,Milestone,2.5.2.,This,binds,the,reports,2047352,2-TN-01[1],and,2047352,2-TN-02[2],as,of,August,27,,2021,,which,are,presented,in,composite,form:,sections,2,and,3,(pp.1-7,of,total,20,pp),comprise,the,original,2047352,2-TN-01.,The,aim,of,this,report,is,to,provide,a,preliminary,description,of,potential,reduction,of,size,of,in-,puts,,size,of,design,,and,size,of,outputs,,and,associated,computational,benefits,of,reduced,order,modelling,for,fusion,codes,,with,a,focus,on,relevant,proxyapps.,Due,to,time,constraints,,a,single,proxyapp,-,describing,anisotropic,heat,transport,-,is,considered,,though,this,is,supported,by,toy,examples,and,further,references,from,the,oeuvre,of,the,authors,(who,are,world,experts,in,the,field),,including,illustrative,examples,of,coupled,models.,The,report,stresses,the,need,for,reduced,order,models,(hereafter,ROMs),in,situations,mandating,large,ensembles,of,expensive,simulations,(as,are,expected,to,arise,in,NEPTUNE,for,e.g.,uncer-,tainty,quantification,or,data,assimilation,aspects).,A,bird’s,eye,perspective,on,the,subject,area,is,provided,and,justification,is,given,for,the,decisions,to,focus,primarily,on,non-intrusive,ROMs,-,meaning,that,there,is,no,need,to,alter,the,subject,simulation,code;,the,ROM,is,then,‘data-driven’,in,that,it,is,constructed,using,solely,the,outputs,of,the,subject,-,a,decision,entirely,consistent,with,the,NEPTUNE,separation,of,concerns,ethos,,and,more,specifically,to,use,the,family,of,Gaussian,process,surrogates,(hereafter,GP-ROM).,Some,relevant,aspects,of,the,current,state-of-the-art,are,indicated,,with,reference,to,the,authors’,recent,works.,The,report,details,a,particular,non-intrusive,ROM,as,applied,to,the,anisotropic,diffusion,problem,(viz.,a,two-dimensional,diffusion,equation,in,which,the,diffusivity,is,represented,by,a,spatially-,varying,anisotropic,tensor).,The,numerical,representation,is,via,the,finite-element,package,Firedrake,,giving,a,solution,comprising,78961,degrees,of,freedom.,This,model,space,dimensionality,is,re-,duced,to,25,using,the,technique,of,proper,orthogonal,decomposition,(POD);,the,reduced,system,is,then,used,to,fit,a,Gaussian,process,using,a,constant,mean,(trend,function),and,a,Mat,´ern,ker-,nel,for,the,prior,covariance.,The,efficacy,of,the,resulting,model,,in,terms,of,predicted,mean,and,confidence,interval,,is,shown,to,be,reasonable,by,comparing,some,of,the,ROM,outputs,to,the,cor-,responding,full,numerical,solutions.,This,section,contains,also,a,technical,exposition,of,the,offline,regression,used,to,construct,the,Gaussian,process,,in,which,the,main,computational,bottleneck,-,a,cubic,scaling,in,the,number,of,data,samples,-,is,emphasized.,A,subsequent,section,provides,a,description,of,active,learning,(AL),for,constructing,a,GP-ROM.,AL,entails,choosing,the,next,sampling,position,dynamically,based,on,the,current,ROM,-,various,the,MacKay,strategy,(giving,ALM),is,to,minimize,the,local,maximum,algorithms,can,be,used:,posterior,predictive,variance,,leading,to,the,problem,of,over-sampling,at,boundaries,,while,the,al-,ternate,Cohn,scheme,(ALC),involves,minimizing,the,same,variance,averaged,over,the,domain,(the,need,for,additional,model,evaluations,during,these,steps,is,avoided,by,a,property,of,Gaussians,under,Bayesian,combination).,The,AL,schemes,are,applied,to,the,anisotropic,diffusion,problem,,demonstrating,that,(1),the,active,learning,saturates,at,large,sample,numbers,giving,there,similar,performance,to,a,static,Latin,hypercube,design,(LHD);,and,(2),the,ALM,performs,worse,than,ALC,or,a,LHD,at,large,sample,number,because,of,the,boundary,over-sampling,problem.,For,realistic,fusion,problems,,the,samples,are,likely,to,be,sufficiently,expensive,that,the,saturated,regime,is,not,encountered,,hence,AL,is,expected,to,be,useful.,The,authors,propose,also,a,strategy,for,identifying,and,excluding,regions,of,flat,response,surface,in,the,offline,phase,,in,order,to,miti-,gate,against,the,cubic,scaling,problem,in,fitting,a,GP-ROM,with,a,large,number,of,samples,-,this,involves,a,dense,exploration,of,the,response,surface,and,then,the,selection,from,these,outputs,of,an,appropriate,set,of,initial,data,,followed,by,active,learning,using,the,data,already,generated,(thus,giving,a,procedure,that,is,explicitly,parallelizable).,This,approach,provides,arguably,some,protection,against,the,problem,of,missing,extreme,outlying,events,-,one,must,bear,in,mind,that,the,quality,of,the,ROM,is,to,a,large,extent,dependent,on,the,experimental,design.,As,a,caution,,however,,it,must,be,noted,that,,for,the,NEPTUNE,use,case,,a,dense,sampling,of,the,response,surface,is,likely,to,represent,a,very,large,computational,challenge.,The,report,closes,with,a,discussion,of,possible,future,directions,in,a,section,in,which,the,recom-,mendations,interspersed,at,appropriate,locations,serve,in,lieu,of,a,formal,concluding,section.,A,proposal,to,test,some,intrusive,methods,in,order,to,mitigate,the,problem,of,excluding,extreme,events,when,applying,dimension,reduction,techniques,(e.g.,POD),is,included;,the,output,here,would,be,a,physics-informed,ROM,(references,in,the,context,of,machine,learning,are,given).,A,subsection,presents,deep,GPs,(meaning,that,the,outputs,of,a,GP,are,used,sequentially,as,the,inputs,to,another,GP),where,the,motivation,is,to,construct,ROMs,for,systems,with,regime,change;,the,deep,GP,is,able,to,incorporate,multiple,kernels,,as,evidenced,by,the,example,of,fitting,a,toy,dataset,containing,distinct,quiescent,and,active,regions.,A,further,subsection,focuses,on,active,subspace,methods,for,reducing,the,input,dimension,(cf.,the,POD,in,Section,3,,which,reduces,the,number,of,internal,model,states).,A,technique,called,sufficient,dimension,reduction,is,briefly,outlined,(with,further,reference,provided),,giving,a,general,method,for,detecting,the,linear,combinations,of,inputs,to,which,a,model,is,most,sensitive.,A,final,subsection,outlines,,with,the,aid,of,a,toy,example,,linked,GPs,,which,differ,from,deep,GPs,in,that,they,model,a,system,that,can,be,explicitly,decomposed,into,component,parts,,the,simpler,components,being,more,amenable,to,GP,emulation,than,is,the,whole,(the,divide-and-,conquer,principle,applies).,This,gives,also,a,semi-intrusive,procedure,for,constructing,a,GP-ROM,for,a,coupled,system.,It,is,explained,that,there,remain,challenges,associated,to,the,dimension,reduction,of,the,intermediate,data,in,cases,where,the,intermediate,states,have,input,dimension,much,higher,than,that,of,the,global,input.,It,is,clear,that,active,sampling,in,the,case,of,a,linked,GP,can,mitigate,the,problem,of,under-exploration,of,the,input,parameter,space,of,intermediate,components,in,a,coupled,model,i.e.,the,fact,that,the,space-filling,property,of,a,LHD,is,lost,once,the,inputs,have,been,propagated,through,one,or,more,component,models,(thus,giving,,amongst,other,problems,,the,potential,to,miss,extreme,events).,2,Acknowledgement,The,support,of,the,UK,Meteorological,Office,and,Strategic,Priorities,Fund,is,acknowledged.,References,[1],D.,Ming,and,S.,Guillas.,Report,on,suitability,and,potential,of,ROM,to,fusion,models,:,A,Non-,intrusive,ROM,for,Solvers,with,High-dimensional,Outputs.,Technical,Report,2047352,2-TN-01,,UKAEA,Project,Neptune,,2021.,[2],D.,Ming,and,S.,Guillas.,Report,on,suitability,and,potential,of,Reduced,Order,Modelling,(ROM),to,fusion,models:,Gaussian,Process,ROM,for,Solvers,with,High-dimensional,Outputs.,Techni-,cal,Report,2047352,2-TN-02,,UKAEA,Project,Neptune,,2021.,UKAEA,REFERENCE,AND,APPROVAL,SHEET,Client,Reference:,UKAEA,Reference:,CD/EXCALIBUR-FMS/0044,Issue:,Date:,1.00,August,27,,2021,Project,Name:,ExCALIBUR,Fusion,Modelling,System,Prepared,By:,Name,and,Department,Wayne,Arter,Ed,Threlfall,Signature,N/A,N/A,Date,August,27,,2021,August,27,,2021,BD,Reviewed,By:,Rob,Akers,August,27,,2021,Advanced,Dept.,Manager,Computing,Approved,By:,Martin,O’Brien,August,27,,2021,MSSC,3,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,Report,on,suitability,and,potential,of,Reduced,Order,Modelling,(ROM),to,fusion,models,Gaussian,Process,ROM,for,Solvers,with,High-dimensional,Outputs,Deyu,Ming,and,Serge,Guillas,University,College,London,Final,report,1,Disclaimer,UKAEA,Report:,2047352,2-TN-02,D1.1,August,2,,2021,We,are,very,grateful,to,Dr,Patrick,Farrell,for,the,provision,of,the,proxyApp,modelling,the,anisotropic,heat,transport,problem.,It,is,the,only,fusion,model,we,could,access,over,the,short,period,of,the,funded,project,(4,January,2021,-,31,July,2021).,We,discussed,with,the,NEPTUNE,team,(Benjamin,Dudson,and,Patrick,Farrell),the,possibility,of,using,another,model,to,couple,two,models,in,a,one-way,coupling,for,UQ,using,ROM:,the,anisotropic,heat,transport,model,and,the,isotropic,heat,conduction,to,the,solid,wall.,But,the,wall,boundary,proxyApp,is,not,yet,available.,As,a,result,,we,could,not,examine,in,practice,the,possibility,of,implementing,ROM,for,UQ,in,the,context,of,nuclear,fusion,modelling,where,models,are,coupled.,We,nevertheless,provided,some,examples,of,UQ,coupling,at,the,end,of,this,report,from,the,paper,Ming,and,Guillas,(2021),and,discussed,future,directions,below.,2,Introduction,Many,modern,physical,computer,models,involve,solving,PDEs,with,numerical,solvers,,such,as,finite,element,methods,(FEM),,which,can,be,computationally,expensive,due,to,•,ever,more,complex,and,larger-scale,models;,•,high-dimensional,input,and,output;,•,large,demands,on,computational,resources.,These,create,challenges,to,efficient,uncertainty,quantification,of,computer,models,,such,as,the,fusion,models,,as,we,often,need,to,run,the,models,many,times,for,tasks,such,as,sensitivity,analysis,,uncertainty,propagation,and,model,calibration.,To,tackle,these,challenges,,reduced,order,models,(ROM),are,needed,to,•,serve,as,low-dimensional,replacements,with,comparable,accuracy;,•,reduce,evaluation,time,of,original,solvers;,•,save,storage,,e.g.,,for,high-dimensional,output.,1,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,Traditional,reduced,order,models,,also,known,as,intrusive,reduced,order,models,,often,are,constructed,using,reduced,basis,methods,(Quarteroni,et,al.,,2015),,among,which,the,Proper,Orthogonal,Decomposition,(POD),is,perhaps,the,most,popular,technique.,The,intrusive,reduced,order,models,for,original,high-fidelity,models,with,high-dimensional,output,are,typically,built,using,a,two-phase,procedure,called,offline-online,decomposition:,•,offline,phase:,high-fidelity,solutions/outputs,are,obtained,and,reduced,basis,is,calculated;,•,online,phase:,the,original,problems,are,projected,onto,the,reduced,space,for,efficient,computation,of,solutions,at,new,inputs.,However,,the,online,phase,of,the,intrusive,reduced,order,modelling,is,challenging,in,practice,because:,•,expertise,and,domain,knowledge,are,required,to,project,the,equations,and,physics,of,the,original,high-fidelity,problems,to,constructed,reduced,space;,•,dimensionality,reduction,techniques,are,largely,constrained,by,the,problem,formulation;,•,uncertainty,is,not,incorporated.,For,these,reasons,,in,this,report,we,focus,on,non-intrusive,reduced,order,models,for,problems,with,high,dimensional,outputs,,utilising,the,family,of,Gaussian,process,(GP),surrogates,(also,known,as,emulators).,GP,emulators,have,been,successfully,implemented,for,dimension,reduction,of,either,outputs,or,inputs.,For,instance:,•,Guillas,et,al.,(2018),used,Functional,Principal,Components,Analysis,(FPCA),as,an,equiv-,alent,approach,to,POD,for,time,series,outputs,of,tsunami,waves,,and,Chang,et,al.,(2019),used,Spherical,Harmonics,and,Gaussian,Markov,Random,Fields,for,optimal,reduction,of,surfaces,outputs.,•,For,inputs,,Liu,and,Guillas,(2017),employed,a,kernel-based,approach,to,extract,the,few,input,field,directions,of,most,influence,for,the,outputs,in,order,to,build,GPs,with,few,input,dimensions,(orders,of,magnitude,gain,in,dimension).,The,report,is,organised,as,follows.,In,Section,3,,a,non-intrusive,ROM,with,GP,surrogates,and,POD,is,described,and,applied,in,a,anisotropic,heat,transport,problem.,We,then,propose,and,discuss,an,active,learning,procedure,to,construct,the,introduced,non-intrusive,ROM,with,an,illustrative,example,in,Section,4.,Future,directions,are,discussed,in,Section,5.,3,Non-intrusive,ROM,with,Gaussian,Process,Surrogates,The,non-intrusive,reduced,order,modelling,is,a,data-driven,approach,that,uses,a,statistical,surrogate,model,to,mimic,the,functional,relations,between,the,model,input,and,constructed,reduced,output,space,in,the,online,phase,of,the,offline-online,decomposition.,The,utilisation,of,statistical,surrogates,alleviates,the,difficulties,involved,in,reformulating,the,original,high-fidelity,problems,under,the,intrusive,reduced,order,modelling.,In,particular,,with,GP,surrogates,we,2,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,are,able,to,quantify,uncertainty,of,the,high-dimensional,outputs,predicted,at,unobserved,input,positions.,Let,X,∈,RN,×D,contain,N,sets,of,D,dimensional,input,of,a,computer,model,,which,produces,N,corresponding,sets,of,K,dimensional,output,Y,∈,RN,×K,accordingly.,Then,,one,can,mimic,the,functional,relationships,between,the,input,X,and,each,output,dimension,Yk,∈,RN,×1,by,a,GP,surrogate,GP,k,independently,for,k,=,1,,.,.,.,,,K,without,considering,the,dependence,between,output,dimensions,(Gu,and,Berger,,2016).,Ignoring,the,potential,cross-dependence,does,not,pose,a,serious,issue,unless,we,are,interested,in,the,joint,distribution,of,the,output,,and,it,can,be,shown,(Kyzyurova,,2017),that,the,independently,constructed,GP,surrogates,correspond,to,the,marginal,GPs,of,a,joint,GP,surrogate,under,certain,dependence,structures.,The,GP,surrogate,GP,k,is,formally,defined,as,a,multivariate,normal,distribution,with,respect,to,Yk:,Yk,∼,N,(µk(X),,σ2,kRk(X)),,in,which,the,i-th,element,of,µk(X),∈,RN,×1,is,often,specified,by,a,trend,function,fk(Xi),with,Xi,∈,R1×D,being,the,i-th,row,of,X,,and,the,ij-th,element,of,Rk(X),∈,RN,×N,is,given,by,ck(Xi,,Xj),,where,ck,is,a,given,kernel,function.,The,trend,function,fk,can,be,formulated,as,a,linear,combination,of,a,set,of,basis,functions,of,Xi,and,we,assume,a,constant,trend,function,fk(Xi),=,bk,in,this,report.,There,are,various,choices,for,ck,(see,Rasmussen,and,Williams,(2006)).,In,this,report,,we,use,the,separable,kernel,function:,ck(Xi,,Xj),=,D,(cid:89),d=1,ck,d(Xid,,Xjd),,where,ck,d,is,a,one-dimensional,kernel,function.,A,typical,choice,for,ck,d,in,computer,model,emulation,is,the,squared,exponential,(SExp),kernel:,ck,d(Xid,,Xjd),=,exp,−,(cid:40),(Xid,−,Xjd)2,γ2,k,d,(cid:41),,,where,γk,d,>,0,is,the,range,parameter.,However,,the,SExp,kernel,has,been,criticised,for,its,over-smoothness,(Stein,,1999),for,physical,problems,as,well,as,its,associated,ill-conditioned,problems,(Dalbey,,2013;,Gu,et,al.,,2018b).,Another,popular,kernel,choice,is,the,Mat´ern,ker-,nel,(Rasmussen,and,Williams,,2006):,ck,d(Xid,,Xjd),=,exp,−,√,(cid:18),2p,+,1,rij,d,γk,d,(cid:19),p!,(2p)!,p,(cid:88),i=0,(p,+,i)!,i!(p,−,i)!,(cid:18),2rij,d,√,2p,+,1,(cid:19)p−i,γk,d,,,where,rij,d,=,Xid,−,Xjd.,The,Mat´ern,kernel,is,known,to,be,less,prone,to,ill-conditioning,issues,and,provides,a,reasonably,adequate,smoothness,to,the,GP,surrogates.,In,particular,,the,Mat´ern-2.5,kernel,,which,is,defined,as,the,Mat´ern,kernel,with,p,=,2:,(cid:32),√,ck,d(Xid,,Xjd),=,1,+,5|Xid,−,Xjd|,γk,d,+,5(Xid,−,Xjd)2,3γ2,k,d,(cid:33),(cid:40),√,exp,−,5|Xid,−,Xjd|,γk,d,(cid:41),,,3,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,is,the,default,kernel,choice,for,many,computer,model,emulation,packages,,such,as,DiceKriging,(Rous-,tant,et,al.,,2012),and,RobustGaSP,(Gu,et,al.,,2018a).,Therefore,,we,employ,the,Mat´ern-2.5,kernel,in,this,report.,The,posterior,predictive,distribution,N,((cid:98)µk(x∗),,(cid:98)σ2,k(x∗)),of,GP,k,with,respect,to,the,output,k,(x∗),at,an,unobserved,input,position,x∗,is,given,in,different,analytical,forms,depending,Y,∗,on,how,the,model,parameters,bk,,σ2,k,and,{γk,d}d=1,...,D,are,estimated.,Different,maximum-,k(x∗),likelihood-based,estimation,approaches,and,the,corresponding,expressions,for,(cid:98)µk(x∗),and,(cid:98)σ2,are,discussed,in,Roustant,et,al.,(2012);,Gu,et,al.,(2018b).,The,main,computational,bottlenecks,of,the,GP,surrogate,construction,are,the,number,of,data,points,N,and,the,dimension,K,of,the,output,of,a,computer,model.,Since,the,inference,of,GP,surrogates,involve,inversions,of,the,correlation,matrix,Rk,∈,RN,×N,with,computational,complexity,of,O(N,3),,it,soon,becomes,computationally,prohibitive,to,build,GP,surrogates,in,practice,when,N,is,more,than,several,thousands.,In,such,a,case,,one,may,need,sparse,approx-,imations,(Liu,et,al.,,2020),to,the,GP,to,reduce,the,computational,complexity,induced,by,the,big,data.,In,computer,model,experiments,,one,often,does,not,have,big,data,(i.e.,,realisations,from,the,underlying,computer,model),due,to,the,limited,computational,budget.,However,,if,the,input,dimension,D,is,large,,then,small,data,are,insufficient,to,explore,adequately,the,whole,input,domain,and,thus,the,resulting,GP,surrogates,can,be,inaccurate.,High,input,dimension,also,causes,challenges,to,the,model,estimation,because,a,large,number,of,range,parameters,{γk,d}d=1,...,D,need,to,be,estimated,for,each,output,dimension.,To,alleviate,this,issue,,one,can,reduce,the,input,dimension,D,to,P,such,that,P,(cid:28),D,by,dimension,reduction,techniques,such,as,POD,,kernel,dimension,reduction,(Liu,and,Guillas,,2017),,and,active,subspace,(Tripathy,et,al.,,2016).,A,high,output,dimension,K,creates,the,issue,that,it,can,be,computational,burdensome,to,build,K,independent,GP,surrogates:,without,parallel,implementation,the,training,and,validation,of,a,huge,amount,of,GP,surrogates,are,practically,infeasible.,This,report,tackles,the,latter,issue,on,high-dimensional,outputs,(e.g.,,a,snapshot,where,each,point,on,the,snapshot,represents,a,FE,solution,and,contributes,to,the,output,dimensionality),produced,by,computer,models.,Perhaps,the,most,straightforward,approach,to,address,the,issue,is,to,reduce,the,output,dimension,K,to,L,such,that,L,(cid:28),K,by,POD.,The,POD,of,Y,∈,RN,×K,can,be,done,with,following,steps:,1.,Compute,the,sample,mean,µY,∈,R1×K,of,Y,and,obtain,the,centred,output,matrix,Yc,=,Y,−,µY;,2.,Implement,the,eigendecomposition,of,G,=,1,c,such,that,G,=,VΛV(cid:62),,where,the,columns,of,V,∈,RN,×N,contains,the,eigenvectors,of,G,and,the,diagonal,of,Λ,∈,RN,×N,contains,the,corresponding,eigenvalues,(λ1,,.,.,.,,,λN,),in,descending,order;,N,YcY(cid:62),3.,Compute,˜V,=,Y(cid:62),N,Y(cid:62),C,=,1,c,Yc;,c,V,∈,RK×N,,,which,contains,the,eigenvectors,of,sample,covariance,matrix,4.,Choose,L,≤,N,and,obtain,the,low,dimensional,output,(cid:98)Y,=,Yc,˜VL,∈,RN,×L,,where,4,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,˜VL,∈,RK×L,contains,the,first,L,eigenvectors,included,in,˜V.,One,can,also,obtain,˜V,by,performing,the,singular,value,decomposition,(SVD),of,Yc,that,is,im-,plemented,,e.g.,,in,the,PCA,function,of,Python,package,scikit-learn,(Pedregosa,et,al.,,2011).,After,obtaining,the,low,dimensional,data,(cid:98)Y,,we,then,construct,L,independent,GP,surrogates,of,each,of,L,dimensions,of,(cid:98)Y.,Let,N,((cid:98)µl(x∗),,(cid:98)σ2,l,(x∗)),be,the,posterior,predictive,distribution,of,(cid:98)Y,∗,l,(x∗),,the,l-th,dimension,of,the,low,dimensional,output,,predicted,at,an,unobserved,input,position,x∗.,Then,the,posterior,predictive,distribution,of,the,corresponding,high,dimensional,output,Y∗(x∗),∈,R1×K,is,given,by,(cid:16),N,(cid:98)µ(x∗),˜V(cid:62),L,+,µY,,˜VL,(cid:98)Σ(x∗),˜V(cid:62),L,(cid:17),,,where,(cid:98)µ(x∗),=,((cid:98)µ1(x∗),,.,.,.,,,(cid:98)µL(x∗)),and,(cid:98)Σ(x∗),=,diag((cid:98)σ2,1(x∗),,.,.,.,,,(cid:98)σ2,L(x∗)).,Figure,1,demonstrates,the,procedure,to,build,non-intrusive,reduced,order,model,with,GP,surrogates.,In,the,offline,phase,,dimension-reduction,techniques,,e.g.,,POD,,are,applied,to,reduce,the,high-dimensional,output,to,a,low-dimensional,space.,Then,in,the,online,phase,,GP,surrogates,are,constructed,independently,on,each,reduced,dimension.,Using,the,constructed,GP,surrogate,and,reduced,basis,,one,can,obtain,the,predicted,low-dimensional,and,in,turn,the,high-dimensional,output,at,new,input,positions,with,little,computational,efforts.,Input,Solver,High-dim,Output,New,Input,GP,Surrogate,Low-dim,Output,Figure,1:,The,workflow,to,construct,non-intrusive,ROM,with,GP.,The,black,arrows,represent,the,offline,phase;,the,blue,arrows,represent,the,online,phase;,the,red,arrows,represent,the,prediction,procedure,using,the,constructed,non-intrusive,ROM,with,GP.,3.1,Example:,2-D,model,of,anisotropic,heat,transport,In,this,section,,we,explore,the,non-intrusive,ROM,with,GP,to,mimic,the,FE,solver,to,the,2-D,problem,“Open,field,lines,with,oscillating,anisotropy,directions”,in,Deluzet,and,Narski,(2019).,The,problem,has,two,key,inputs,m,and,α,that,control,the,anisotropy,of,the,solution,field,,i.e.,,the,anisotropy,direction,is,defined,by,b,=,B,|B|,,,B,=,(cid:32),α(2y,−,1),cos(mπx),+,π,παm(y2,−,y),sin(mπx),(cid:33),,,where,m/2,is,the,number,of,oscillation,periods,in,the,computational,domain,and,α,is,the,amplitude.,The,output,is,a,high-dimensional,2-D,field,defined,on,the,square,computational,domain,[0,,1],×,[0,,1],and,allows,a,closed,form,solution.,5,164,3.1.1,Experimental,Setup,165,166,167,168,169,170,171,172,173,174,175,To,construct,the,reduced,basis,via,the,POD,and,the,GP,surrogate,,N=40,samples,are,arranged,in,a,Latin,hypercube,over,m,∈,[0,,12],and,α,∈,[0,,3],(see,the,left,plot,in,Figure,2).,We,then,run,the,FE,solver,(implemented,in,Firedrake,(Rathgeber,et,al.,,2016)),of,the,toy,problem,to,obtain,the,corresponding,2-D,outputs,,each,of,which,contains,FE,solutions,on,K,=,78961,nodes.,These,40,×,78961,high-dimensional,outputs,are,then,reduced,to,40,low-dimensional,outputs,(40,×,25),using,POD,by,retaining,the,first,25,principal,components,out,of,the,total,40,components,,see,the,right,plot,in,Figure,2,,where,the,cumulative,explained,variance,is,defined,as,be,the,number,of,components.,(cid:80)L,(cid:80)N,i=1,λi,i=1,λi,with,L,Figure,2:,(Left):,Training,and,designing,points,generated,for,the,inputs,m,and,α.,The,blue,points,are,design,input,locations,generated,from,the,Latin,hypercube,design,and,the,red,points,are,testing,input,locations;,(Right):,cumulative,explained,variance,given,by,the,POD.,GP,surrogates,are,then,constructed,independently,for,each,of,the,25,dimensions,of,the,reduced,order,data.,GP,surrogates,are,trained,with,the,Mat´ern-2.5,kernel,using,the,RobustGaSP,package,in,R.,176,3.1.2,Experimental,Results,177,178,179,180,181,182,183,184,185,186,187,188,189,190,We,test,the,constructed,non-intrusive,ROM,at,four,testing,input,positions,(m,,α),=,(6,,2),,(10,,2),,(1,,2),and,(10,,0),(see,the,left,plot,of,Figure,2).,The,FE,solutions,(from,the,Firedrake),and,the,predicted,solutions,from,the,built,ROM,are,compared,in,Figure,3.,The,normalised,(to,the,range,of,FE,solutions),errors,between,the,FE,solutions,and,the,predicted,solutions,from,the,built,ROM,are,shown,in,Figure,4.,The,coverage,of,the,ROM,(i.e.,,the,instances,that,the,FE,solutions,fall,within,the,predictive,bounds,of,GP-based,ROM),are,also,given,in,Figure,5.,It,can,been,seen,from,these,results,that,the,constructed,ROM,with,GP,could,predict,well,the,FE,solutions,of,the,anisotropic,problem,at,input,locations,that,are,not,realised.,Among,the,four,testing,positions,,the,final,case,with,m,=,10,and,α,=,0,presents,the,largest,normalised,errors,up,to,13%.,This,is,not,a,surprising,result,because,m,has,no,effect,on,the,FE,solution,of,the,problem,when,α,=,0.,However,,this,information,is,not,fully,captured,in,the,training,data,and,thus,not,gained,by,the,non-intrusive,ROM,with,GP,,which,is,pure,data-driven,method,that,only,understands,the,functional,relation,between,m,,α,and,the,solution,field,from,the,training,set.,As,a,result,,we,could,observe,5,blurred,oscillation,periods,in,the,predicted,solutions,from,6,Figure,3:,Comparisons,of,FE,solutions,to,the,predicted,solutions,given,by,the,constructed,GP-,based,ROM.,The,first,row,gives,the,FE,solutions.,The,second,row,gives,the,predicted,solutions,from,the,GP-based,ROM.,The,columns,from,left,to,right,correspond,to,testing,input,positions,(m,,α),=,(6,,2),,(10,,2),,(1,,2),and,(10,,0),respectively.,Figure,4:,The,normalised,errors,between,FE,solutions,and,the,predicted,solutions,from,the,ROM,with,GP,surrogate.,The,plots,from,left,to,right,correspond,to,testing,input,positions,(m,,α),=,(6,,2),,(10,,2),,(1,,2),and,(10,,0),respectively.,ROM,in,Figure,3.,However,,the,predictive,interval,(whose,upper,and,lower,bounds,are,given,at,two,standard,deviations,2(cid:98)σ,above,and,below,the,predictive,mean,(cid:98)µ),of,the,GP-based,ROM,covers,the,FE,solutions,sufficiently,in,this,case,,demonstrating,that,one,can,benefit,from,the,predictive,uncertainty,embedded,in,the,non-intrusive,ROM,coupled,with,GP,emulation.,4,Active,learning,for,Non-intrusive,ROM,with,Gaussian,Pro-,cess,Surrogates,4.1,Why,Active,Learning?,Active,learning,,also,known,as,sequential,design,,is,a,collection,of,approaches,that,adaptively,enrich,the,training,points,for,surrogate,modelling,of,computer,solvers.,In,comparison,to,one-,7,191,192,193,194,195,196,197,198,199,Figure,5:,The,coverage,of,constructed,ROM,with,GP,,giving,the,instances,that,FE,solutions,fall,within,the,predictive,bounds,provided,by,the,ROM,with,GP.,1,indicates,that,the,FE,solution,is,covered,by,the,predictive,interval,(whose,upper,and,lower,bounds,are,given,at,two,standard,deviations,2(cid:98)σ,above,and,below,the,predictive,mean,(cid:98)µ),and,0,indicates,otherwise.,The,plots,from,left,to,right,correspond,to,testing,input,positions,(m,,α),=,(6,,2),,(10,,2),,(1,,2),and,(10,,0),respectively.,shot,designs,,such,as,Latin-hypercube,designs,(LHD),,the,active,learning,is,preferred,in,many,cases:,•,One,wants,a,proper,utilisation,of,computational,resources.,Active,learning,allows,one,to,choose,computer,model,input,locations,adaptively,,and,therefore,can,monitor,the,quality,of,the,resulting,surrogate,model,while,the,active,learning,is,in,progress,and,determine,whether,to,pause,or,continue,the,model,evaluations;,•,More,computer,model,evaluations,are,needed,in,the,input,region,of,interest.,Unlike,static,space-filling,designs,,such,as,LHD,,active,learning,,depending,on,the,quality,of,the,under-,lying,surrogate,model,(as,we,will,discuss,in,Section,4.4),,could,direct,the,computer,models,to,evaluate,at,input,locations,where,the,model,response,exhibits,more,variations,and,thus,are,more,of,interest;,•,There,are,existing,computer,model,evaluations,,but,are,potentially,large,in,size,and/or,not,produced,with,a,careful,design.,It,can,be,computationally,inefficient,to,generate,a,new,design,,e.g.,,a,static,space-filling,design,,if,one,has,an,existing,set,of,model,evaluations,because,one,could,utilise,the,data,available.,However,,it,can,be,both,numerically,inefficient,(e.g.,,the,design,formed,by,the,existing,data,is,poor),and,computationally,burdensome,(e.g.,,the,existing,data,is,of,large,size),to,use,the,whole,existing,model,realisations,for,surrogate,modelling.,Thus,,one,can,use,active,learning,to,choose,training,data,adaptively,from,the,existing,model,evaluations,from,a,small,design,size,while,at,the,same,time,prevent,from,the,numerical,instabilities,induced,by,poor,designs;,•,There,is,a,system,of,coupled,computer,models.,It,has,been,shown,in,Ming,and,Guillas,(2021),that,active,learning,is,essential,to,construct,Gaussian,process,(GP),based,surrogate,models,in,a,computationally,efficient,and,effective,manner.,Static,designs,of,global,inputs,can,produce,poor,designs,,and,thus,numerical,issues,,to,sub-models,of,a,computer,system,,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,8,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,and,can,also,waste,computational,resources,over,input,regions,of,sub-models,that,are,not,contributing,to,the,global,outputs,(that,correspond,to,the,global,input,region,of,interest).,4.2,Implementation,Assume,that,we,have,data,Dn,=,{Xn,,Yn},that,consists,of,input,Xn,∈,Rn×D,and,the,responding,high-dimensional,computer,model,output,Yn,∈,Rn×K.,Then,,a,generic,active,learning,procedure,that,selects,the,next,input,position,xn+1,to,be,evaluated,by,the,computer,model,for,refinement,of,GP,based,non-instrusive,ROM,(abbreviated,as,GP-ROM,in,the,remainder,of,the,report),introduced,in,Section,3,is,given,in,Algorithm,1.,Once,xn+1,is,determined,,one,can,then,obtain,the,augmented,data,Dn+1,=,{Xn+1,,Yn+1},by,concatenating,xn+1,and,its,corresponding,high-,dimensional,output,yn+1,to,Dn,and,update,GP-ROM,{GP,l},by,re-invoking,Algorithm,1.,Algorithm,1,Active,learning,for,GP-ROM,Input:,(i),Dn,=,{Xn,,Yn};,(ii),a,candidate,set,C,of,input,locations,{xi}i=1,...,M,.,Output:,The,next,input,position,xn+1,to,be,evaluated,by,the,computer,model.,1:,Compute,the,low-dimensional,output,(cid:98)Yn,∈,Rn×L,of,Yn,and,the,corresponding,eigenvalues,λl=1,...,L,using,POD;,2:,Construct,GP-ROM,{GP,l},using,{Xn,,(cid:98)Yn};,3:,Calculate,the,criterion,Il(x),at,each,input,locations,in,C,using,GP,l,for,all,l;,4:,Choose,for,the,next,input,position,xn+1,by,solving,xn+1,=,argmax,x∈C,L,(cid:88),l=1,wlIl(x),with,wl,=,λl,i=1,λi,(cid:80)n,We,present,two,candidates,for,the,criterion,Il(x),based,on,the,Active,Learning,MacKay,(ALM),(MacKay,,1992),and,the,Active,Learning,Cohn,(ALC),(Cohn,,1996),respectively,for,the,selection,of,xn+1.,ALM,aims,to,find,the,next,input,location,that,corresponds,to,the,maximum,predictive,variance,exhibited,by,the,GP-ROM.,Thus,,Il(x),is,defined,by,Il(x),=,(cid:98)σ2,l,(x),,where,(cid:98)σ2,l,(x),is,the,posterior,predictive,variance,of,GP,l,evaluated,at,x.,However,,ALM,has,a,well-know,issue,that,it,selects,excessive,input,locations,around,boundaries,of,the,input,region,because,of,the,lack,of,data,beyond,boundaries.,To,alleviate,this,issue,,ALC,aims,to,select,the,input,position,such,that,the,integrated,predictive,variance,of,GP-ROM,over,the,input,region,is,minimised,after,augmenting,xn+1,to,Xn.,Formally,,Il(x),under,ALC,is,defined,by,Il(x),=,−,(cid:90),x∗∈X,(cid:16),x∗|[X(cid:62),n,,,x(cid:62)](cid:62)(cid:17),dx∗.,(cid:98)σ2,l,l,(cid:0)x∗|[X(cid:62),n,,,x(cid:62)](cid:62)(cid:1),is,interpreted,as,the,posterior,predictive,variance,of,GP,l,evaluated,at,where,(cid:98)σ2,x∗,given,the,input,data,Xn,being,augmented,by,xn.,It,is,worth,noting,that,the,computation,of,n,,,x(cid:62)](cid:62)(cid:1),does,not,require,evaluations,of,the,associated,computer,model,at,xn,because,(cid:98)σ2,the,predictive,variance,of,GP,does,not,depend,on,the,output,data.,In,practice,,the,integral,(cid:0)x∗|[X(cid:62),l,involved,in,ALC,can,be,approximated,by,the,Monte,Carlo,integration,over,a,reference,set,X,9,239,240,241,242,243,244,245,246,(that,can,be,the,same,as,the,candidate,set,C),generated,by,the,LHD.,To,implement,a,full,active,learning,procedure,,one,often,starts,with,a,small,data,set,that,is,generated,by,a,static,design,,such,as,LHD,,and,then,execute,T,iterations,of,Algorithm,1,to,enrich,the,initial,data,set,with,T,additional,realisations,from,the,computer,model.,4.3,Active,learning,for,the,GP-ROM,emulation,of,the,2-D,anisotropic,heat,transport,model,In,this,section,,we,demonstrate,how,efficiency,gains,can,be,made,using,active,learning,for,the,GP-ROM,of,the,FE,solver,to,the,2-D,problem,described,in,Section,3.1,247,4.3.1,Experimental,Setup,To,initiate,the,active,learning,to,build,GP-ROM,,N=20,initial,training,data,points,,whose,input,locations,are,generated,via,the,LHD,over,m,∈,[0,,12],and,α,∈,[0,,3],with,the,corresponding,2-D,output,(that,contains,K,=,78961,solution,nodes),determined,by,running,the,FE,solver,(implemented,in,Firedrake,(Rathgeber,et,al.,,2016)).,We,then,iterate,Algorithm,1,for,both,ALM,and,ALC,80,times,to,augment,additional,80,training,data,points,to,the,initial,data,set.,At,each,iteration,of,the,active,learning,,we,choose,the,number,of,components,L,(in,Line,1,of,Algorithm,1),to,be,retained,from,POD,based,on,the,following,criteria:,L,=,argmin,L∗∈{1,...,n},(cid:12),(cid:12),(cid:12),(cid:12),(cid:12),(cid:80)L∗,(cid:80)n,i=1,λi,i=1,λi,(cid:12),(cid:12),(cid:12),−,0.9998,(cid:12),(cid:12),,,where,λ1,>,λ2,>,·,·,·,>,λn.,To,take,into,account,the,effects,of,initial,data,set,on,the,active,learning,,we,repeat,both,ALM-,and,ALC-based,active,learning,10,times.,For,the,comparison,between,ALM,and,ALC,,we,generate,2500,testing,data,points,over,m,∈,[0,,12],and,α,∈,[0,,3],and,compute,the,Normalised,Root,Mean,Squared,Error,(NRMSE),at,each,active,learning,iteration,by,NRMSE,=,1,2500,2500,(cid:88),i=1,(cid:113),1,K,((cid:101)zi,−,zi)((cid:101)zi,−,zi)(cid:62),max(zi),−,min(zi),×,100%,,248,249,250,251,where,(cid:101)zi,∈,R1×K,and,zi,∈,R1×K,are,2-D,FE,solution,fields,generated,by,the,GP-ROM,and,Firedrake,at,the,i-th,testing,input,location,,respectively.,In,terms,of,implementation,,we,construct,GP-ROM,and,compute,corresponding,ALM,and,ALC,criterions,at,each,iteration,of,the,active,learning,using,the,laGP,package,in,R.,252,4.3.2,Experimental,Results,253,254,255,256,257,258,Figure,6,presents,the,NRMSEs,of,GP-ROMs,built,with,ALM-,and,ALC-based,active,learning,over,80,iterations,,in,comparison,to,those,constructed,with,the,static,LHD,at,various,design,sizes.,It,can,be,observed,that,for,design,size,less,than,50,,GP-ROMs,trained,using,the,active,learning,,regardless,of,ALM,or,ALC,,provide,higher,accuracy,than,those,trained,using,the,static,LHD.,However,,as,the,design,sizes,increases,,the,accuracy,of,GP-ROMs,built,by,the,active,learning,and,LHD,are,comparable.,This,is,because,with,a,large,design,size,,the,input,domain,is,10,259,260,261,262,263,264,265,266,densely,space-filled,by,the,LHD,and,thus,the,NRMSE,of,the,corresponding,GP-ROM,converges,to,that,of,the,GP-ROM,trained,with,the,active,learning.,Figure,6:,Comparison,of,NRMSEs,of,GP-ROM,constructed,using,the,ALM-based,active,learn-,ing,,the,ALC-based,active,learning,,and,the,static,LHD.,We,also,observe,from,Figure,6,that,for,design,size,larger,than,60,GP-ROMs,constructed,by,LHD,perform,better,(in,terms,of,overall,lower,NRMSE),than,those,built,by,ALM-based,active,learning.,This,observation,can,be,explained,by,the,fact,that,ALM-based,active,learning,has,the,tendency,to,choose,excessive,input,locations,around,boundaries,of,the,input,domain,(see,Figure,7(a)),and,thus,could,fail,to,achieve,a,satisfactory,design,,in,which,input,locations,are,preferred,to,be,scattered,within,the,input,domain,of,interest,(see,Figure,7(b)).,(a),ALM,(b),ALC,Figure,7:,Designs,produced,by,a,random,trial,(out,of,10,repeated,trials),of,ALM-,and,ALC-based,active,learning.,267,4.4,Discussion,268,269,270,In,this,section,,we,introduce,a,simple,and,effective,procedure,to,implement,the,active,learning,for,GP-ROM,construction.,Although,the,active,learning,may,eventually,produce,a,space-filling,design,,it,gives,the,computer,model,experimenters,more,controls,over,their,computational,re-,11,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,sources.,One,may,criticise,that,active,learning,is,not,computationally,efficient,in,the,sense,that,it,directs,model,runs,sequentially,and,thus,can,be,time-consuming,in,comparison,to,static,one-,shot,designs,in,which,model,runs,can,be,done,in,parallel.,This,statement,is,sensible,when,one,posses,sufficient,computational,power,(for,parallel,computing),and,active,learning,also,produces,a,space-filling,design.,However,,in,real-world,data,these,conditions,may,not,be,fulfilled.,Our,computational,resources,may,not,permit,us,to,obtain,model,realisations,that,cover,adequately,the,input,region,of,interest,(for,an,accurate,surrogate,model),and,a,space-filling,design,may,not,capture,sufficiently,(without,tremendous,computational,efforts),the,input,regions,where,the,model,response,exhibits,abrupt,changes,,even,if,we,have,an,advanced,surrogate,model,(that,is,suitable,for,both,stationary,and,non-stationary,data).,On,the,contrary,,active,learning,has,the,ability,to,focus,on,input,regions,where,the,corresponding,output,surfaces,show,more,variations,,given,that,the,underlying,surrogate,model,provides,a,satisfactory,uncertainty,quantification,(e.g.,,highlighting,the,regions,with,higher,predictive,standard,deviations).,A,fact,often,forgotten,in,computer,model,experiments,is,that,design,and,surrogate,modelling,are,not,two,separate,tasks.,Good,designs,produce,good,surrogates,with,less,numerical,issues,and,more,reliable,uncertainty,quantification,,which,in,turn,induces,designs,that,better,represent,the,functional,behaviours,of,computer,models,under,the,consideration.,These,are,the,reasons,why,active,learning,could,be,preferred,to,static,space-filling,designs,,which,could,cause,the,surrogate,modelling,challenging,(e.g.,,a,large,number,of,realisations,that,are,needed,to,capture,well,the,computer,model,can,cause,the,GP-ROM,computationally,prohibitive),and,do,not,utilise,the,uncertainties,quantified,by,surrogate,models,for,design,improvement.,It,is,worth,noting,that,active,learning,does,not,guarantee,the,locations,of,(possibly,very,small,but,important),input,regions,of,a,computer,model,that,correspond,to,abrupt,changes,to,the,model,responses.,The,design,produced,by,the,active,learning,depends,on,the,quality,of,the,underlying,surrogate,model,,which,in,turn,depends,on,the,information,contained,in,the,training,data,(assuming,that,the,surrogate,represents,the,training,data,adequately,and,produces,sensible,uncertainty,quantification).,Therefore,,whether,active,learning,could,find,input,regions,that,has,very,localised,and,important,features,depends,on,if,the,information,of,the,regions,exists,in,the,training,data.,For,this,reason,,it,is,vital,to,have,a,good,initial,design,that,incorporates,such,information,for,the,active,learning.,However,,in,practice,this,can,be,difficult,to,achieve,,particularly,for,high-dimensional,cases,,even,we,have,some,prior,knowledge,that,such,non-stationary,features,exist,in,the,computer,model,,and,as,a,consequence,we,may,obtain,a,surrogate,that,completely,ignores,these,regions,with,significant,computational,costs,being,wasted.,To,alleviate,this,issue,,one,could,simply,evaluate,the,computer,model,with,a,high-,resolution,design,using,the,parallel,computing.,In,this,way,,the,local,behaviours,of,a,computer,model,can,be,captured,within,a,reasonable,amount,of,time.,Nevertheless,,it,is,not,advisable,to,use,all,model,evaluations,for,surrogate,modelling,,especially,for,GP-based,surrogates,because,the,large,amount,of,data,can,cause,GP,surrogates,computationally,prohibitive,and,some,evaluations,(e.g.,,that,form,a,flat,response,surface),are,redundant,for,surrogate,improvement.,As,a,result,,we,propose,the,following,hybrid,static-active,learning,procedure,to,address,the,scenario,in,which,we,aim,to,construct,efficiently,(in,terms,of,computation,and,time),a,surrogate,model,that,could,mimic,the,underlying,computer,model,with,localised,behaviours:,12,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,1.,Generate,a,data,set,by,evaluating,the,computer,model,over,a,dense,space-filling,design,in,parallel;,2.,Choose,a,subset,of,the,produced,data,set,as,the,initial,design,for,the,active,learning;,3.,Implement,the,active,learning,that,adaptively,refines,the,design,and,the,surrogate,model,,e.g.,,GP-ROM,,by,selecting,data,points,from,the,data,set,produced,in,Step,1.,There,are,several,benefits,provided,by,the,above,procedure.,Firstly,,the,high-resolution,design,provides,some,guarantees,that,our,data,contain,information,of,localised,behaviours,embedded,in,the,underlying,computer,model.,In,addition,,unlike,typical,active,learning,that,evaluates,models,sequentially,,active,learning,in,Step,3,uses,the,data,set,already,generated,with,a,parallelisable,strategy,and,thus,could,save,a,considerate,amount,of,time,(especially,when,computer,models,are,very,expensive,to,run).,Furthermore,,with,active,learning,one,is,able,to,pick,(potentially,a,small,amount,of),data,points,(from,the,generated,data,set),that,contribute,most,to,the,surrogate,quality,,instead,of,naively,pouring,the,whole,data,set,into,the,surrogate,construction,(causing,computational,difficulties).,Perhaps,the,most,decisive,and,challenging,step,of,the,above,procedure,is,Step,2,because,,as,discussed,,one,expects,to,incorporate,some,information,of,localised,behaviours,of,a,computer,model,into,the,initial,design,such,that,the,resulting,surrogate,is,less,likely,to,overlook,these,features.,How,to,integrate,experts’,knowledge,about,the,localised,features,into,the,initial,design,is,worth,exploring,in,the,future,,but,the,procedure,above,indicates,a,potentially,brutal,but,simply,implementation,for,Step,2:,choose,multiple,random,subsets,of,the,data,set,,then,proceed,to,Step,3,for,multiple,surrogate,constructions,,and,choose,the,surrogate,that,gives,the,best,predictive,accuracy,(e.g.,,lowest,overall,predictive,error,against,the,generated,data,set).,This,implementation,is,computationally,efficient,because,active,learnings,in,Step,3,initiated,by,different,random,designs,can,be,executed,in,parallel,and,do,not,involve,computer,model,evaluations.,5,Future,Directions,We,demonstrate,in,this,report,that,a,GP-ROM,could,be,used,to,replace,computationally,expen-,sive,computer,solvers,for,problems,with,high-dimensional,output,,in,one,of,the,building,blocks,of,nuclear,fusion,modelling.,However,,dimension,reduction,techniques,such,as,POD,lose,informa-,tion,when,the,original,data,are,projected,onto,a,lower,dimensional,space,,and,thus,some,extreme,but,important,events,could,be,masked,in,the,low,dimensional,data,,a,scenario,called,masking,effect.,As,a,result,,if,the,surrogate,is,built,on,the,low,dimensional,data,one,may,not,be,able,to,recover,these,outlying,events,using,the,constructed,non-intrusive,ROM.,Therefore,,other,dimen-,sion,reduction,methods,that,may,be,more,resistant,to,the,masking,effect,could,be,examined.,In,addition,,although,GP-ROM,requires,no,domain,knowledge,and,access,to,the,source,code,of,original,problems,,it,ignores,the,physics,implied,by,the,underlying,problem,and,thus,may,be,inaccurate,comparing,to,the,its,intrusive,counter-party.,Therefore,,it,would,be,worth,exploring,the,trade-off,between,the,speed,and,accuracy,of,intrusive,and,non-intrusive,MOR,,especially,in,context,of,UQ.,It,would,also,be,interesting,to,find,a,middle,ground,where,one,could,exploit,the,benefits,(e.g.,,accuracy,,speed,and,uncertainty),of,both,intrusive,and,non-intrusive,ROM,,13,352,353,354,355,356,357,producing,a,physics-informed,non-intrusive,ROM.,Some,relevant,literature,on,physics-informed,machine,learning,(say,using,a,boundary,condition,or,other,approaches),include,Vernon,et,al.,(2019);,Kashinath,et,al.,(2021);,Watson-Parris,(2021).,Recommendation:,Investigate,how,to,apply,physics-informed,GP-ROM,in,key,nuclear,fusion,models.,Examine,how,to,build,new,types,of,GP-ROM,for,the,case,of,particle-based,models,(PIC),whose,outputs,need,to,be,understood,as,a,continuum.,358,5.1,Deep,GP,for,Non-intrusive,ROM,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,In,this,report,we,explored,how,to,construct,GP-ROM,using,active,learning.,Active,learning,is,particularly,useful,when,the,underlying,computer,model,exhibits,non-stationary,features,as,it,has,the,ability,to,produce,a,non-uniform,design,that,appreciates,the,non-stationarity.,However,,the,success,of,the,active,learning,relies,on,the,quality,of,uncertainty,quantified,by,the,surrogate,model.,Since,conventional,GP,surrogates,assume,stationarity,,more,advanced,non-stationary,GP,models,,such,as,deep,Gaussian,processes,(Damianou,and,Lawrence,,2013),,would,be,good,candidates,for,non-intrusive,ROM,of,fusion,models,that,exhibits,non-stationarity.,Deep,Gaussian,processes,(DGPs),are,feed-forward,compositions,of,conventional,stationary,GPs,with,flexible,model,expressiveness,,particularly,for,non-stationary,data.,However,,training,and,prediction,of,DGP,based,emulators,are,challenging,due,to,the,non-linearity,induced,by,the,kernel,functions,involved,in,GPs.,Various,inference,methods,thus,are,introduced,to,tackle,this,issue.,Variational,inferences,,such,as,Doubly,Stochastic,Variational,Inference,(DSVI),(Salimbeni,and,Deisenroth,,2017),,is,computationally,thrifty,but,is,not,accurate,because,simplified,assumptions,over,the,latent,variables,in,DGP,hierarchy,are,assumed.,On,the,contrary,,the,fully-Bayesian,approach,introduced,by,Sauer,et,al.,(2020),gives,a,comprehensive,uncertainty,quantification,of,DGPs,,but,at,the,expense,of,computation.,The,stochastic,imputation,approach,recently,proposed,by,Ming,et,al.,(2021),is,a,DGP,inference,method,that,enjoys,both,computational,speed,and,the,predictive,accuracy,,and,could,be,a,competitive,and,potential,candidate,for,DGP,emulations,of,non-stationary,fusion,models.,It,is,implemented,in,the,dgpsi,package1.,Figure,8,showcases,the,ALM-based,active,learning,using,a,two-layered,DGP,surrogate,(i.e.,,composition,of,two,stationary,GPs),trained,with,the,stochastic,imputation,in,comparison,to,that,using,a,stationary,GP.,It,can,be,observed,that,DGP,surrogate,outperforms,the,GP,surrogate,in,both,mean,predictions,and,uncertainty,quantification.,In,addition,,with,DGP,the,active,learning,could,produce,a,non-uniform,design,that,appreciates,the,non-stationarity,of,the,underlying,data.,While,the,active,learning,essentially,produces,a,quite,space-filling,design,under,GP,,it,assigns,three,time,more,number,of,design,points,to,the,rough,(and,more,interesting),regime,over,[0,,0.5],than,the,flat,regime,over,(0.5,,1],under,DGP.,Although,this,is,a,simple,1-D,example,,it,gives,motivations,why,DGP,surrogate,should,be,seriously,considered,if,the,reduced-order,output,of,a,fusion,model,is,non-stationary,and,the,active,learning,is,employed.,Recommendation:,Investigate,how,to,reduce,dimensionality,of,outputs,for,key,nuclear,fusion,models,whose,behaviour,may,present,sharp,transitions,or,various,regimes,,such,as,turbulence,models.,The,key,question,is,then,how,to,understand,and,represent,the,continuum,of,outputs,1https://github.com/mingdeyu/DGP,14,(a),GP,(b),DGP,Figure,8:,ALM-based,active,learning,using,GP,and,DGP,emulations.,Solid,line,represents,the,underlying,true,function;,Dashed,line,is,the,mean,prediction;,Shaded,area,represents,95%,predictive,interval;,Dots,(6,in,total),are,initial,training,points,and,triangles,(14,in,total),are,training,points,enriched,by,the,active,learning,procedure,using,GP,and,DGP,surrogates.,The,vertical,dashed,line,indicates,a,visual,split,of,the,underlying,true,function,into,a,rough,regime,over,[0,,0.5],and,a,flat,regime,over,(0.5,,1].,391,392,393,features,across,regimes.,Indeed,these,features,shown,in,3,can,vary,across,regimes,and,must,be,made,consistent,by,some,form,of,joint,augmentation,possibly,at,a,small,cost,but,with,large,benefits,for,emulation.,394,5.2,Active,subspace,for,efficient,dimension,reduction,of,inputs,395,396,397,398,399,400,401,402,403,404,405,406,The,efficiency,of,reducing,dimensions,in,the,inputs,was,demonstrated,in,Liu,and,Guillas,(2017).,Gains,of,orders,of,magnitude,can,be,achieved.,For,instance,,the,application,to,a,surface,of,inputs,(a,mesh,of,3200,elements),enabled,a,reduction,from,dimension,3200,to,5,with,fast,and,accurate,emulation.,Only,about,100,simulations,were,needed,to,come,up,with,5,key,dimensions,as,a,recombination,of,the,original,3200,dimensions.,A,summary,of,the,method,is,presented,below.,It,is,implemented,in,the,Alan,Turing,Institute,Package,Multi-Output,Gaussian,Process,Emulator,(MOGP)2.,The,context,is:,•,Simulator,input,X,(high,dimension,Rp),and,output,Y,=,f,(X),(one,dimension,R1),•,GP,emulation:,fit,an,GP,and,predict,f,(xnew),using,a,sample,of,simulations,f,(X1),,...,,f,(Xn),•,Find,a,reduced,space,(known,as,sufficient,dimension,reduction,SDR),R(X),∈,Rd,,d,<,p,,such,that,there,is,(nearly),no,loss,of,information,in,predicting,Y,by,providing,R(X),instead,of,X,•,To,achieve,SDR,,employ,the,gradient-based,Kernel,Dimension,Reduction,(gKDR),ap-,proach,(Fukumizu,and,Leng,,2014):,2https://github.com/alan-turing-institute/mogp-emulator,R(X),=,BT,X,,BT,B,=,Id,,d,<,p.,15,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,Estimate,B,from,simulation,samples,(X1,,Y1),,...,,(Xn,,Yn).,Note,that,no,strong,assumption,are,made,on,the,variables,(type,,distribution,,dimension).,The,specific,technical,steps,in,gKDR,involve,two,Reproducing,kernel,Hilbert,spaces,(RKHS):,•,Prepare,kernels,kX,and,kY,,,with,the,associated,(RKHS),HX,and,HY,•,The,quantities,of,interest,are,the,gradients,∂E[g(Y,)|X],for,any,g,∈,HY,as,their,evaluation,is,the,ingredient,for,the,identification,of,the,reduced,subspace,,by,looking,at,the,most,∂X,influential,directions.,•,Estimate,(see,Fukumizu,and,Leng,(2014),for,details),ˆMn,=,1,n,n,(cid:88),i=1,∇kX,(Xi)T,(GX,+,n(cid:15)nI)−1GY,(GX,+,n(cid:15)nI)−1∇kX,(Xi),where,GX,and,GY,are,the,Gram,matrices,(kX,(Xi,,Xj)),and,(kY,(Yi,,Yj)),,and,∇kX,(x),=,(∂kX,(X1,,x)/∂x,,...,,∂kX,(Xn,,x)/∂x)T,∈,Rn×m,for,any,x,∈,Rm.,•,Eigen-decompose,ˆMn,into,ˆMn,=,ˆQˆΛ,ˆQT,and,partition,(cid:34)ˆΛ1,ˆΛ,=,(cid:35),,,ˆΛ2,ˆQ,=,[,ˆB,ˆC],,where,ˆΛ1,=,diag(ˆλ1,,...,,ˆλd),consisting,of,the,first,d,largest,eigenvalues,,to,ultimately,provide,the,dimension,reduction.,The,emulation,with,dimension,reduction,can,be,carried,out,and,its,loss,quantified,(Liu,and,Guillas,,2017):,•,f,(X),≈,ˆf,(,ˆBT,X),(cid:13),(cid:13),(cid:13)f,−,ˆf,(cid:13),(cid:13),(cid:13)L2,,=,Op,,4,λd,−,λd+1,n−,min{,1,3,,,2β+1,4β+4,},(cid:32),d,(cid:88),i=1,(cid:33),1,2,ciˆλ2,i,+,(cid:32),m,(cid:88),(cid:33),1,2,,,ciˆλ2,i,i=d+1,•,Build,emulator,˜f,≈,ˆf,on,low,dimensional,space,ˆBT,X,•,Approximation,procedure:,f,(X),≈,ˆf,(,ˆBT,X),≈,˜f,(,ˆBT,X),The,choice,of,remained,dimension,d,and,hyperparameters,is,performance,based,(e.g.,in,the,quality,of,the,predictions,in,a,leave-one-out,strategy),and,can,result,in,very,large,gains,(Liu,and,Guillas,,2017).,Recommendation:,Investigate,how,to,reduce,dimensionality,of,inputs,of,key,nuclear,fusion,models,such,as,the,magnetic,field,modelled,input,of,the,anisotropic,heat,transfer,model.,16,429,5.3,Linked,GP,for,Non-intrusive,ROM,430,431,432,433,434,435,436,437,438,439,440,441,442,Since,fusion,models,are,often,multi-disciplinary,and,multi-physics,,the,recent,advances,on,linked,Gaussian,process,surrogates,(Ming,and,Guillas,,2021),must,be,considered.,The,linked,GP,is,implemented,in,the,dgpsi,package3.,As,an,illustration,,consider,a,toy,system,that,consists,of,two,feed-forward,connected,computer,models,shown,in,Figure,9.,By,directly,applying,con-,ventional,GP,emulation,,one,fails,to,capture,the,local,feature,(over,[−1,,1]),of,the,underlying,system,with,ten,system,runs,,see,Figure,10(d).,However,,if,the,linked,GP,is,employed,to,the,system,by,constructing,GP,surrogates,on,sub-models,individually,with,active,learning,(see,Fig-,ure,10(a),and,10(b)),,one,could,capture,the,local,feature,of,the,overall,system,sufficiently,(see,Figure,10(c)).,This,is,because,the,local,feature,of,the,entire,system,over,[−1,,1],is,created,by,the,composition,of,simpler,individual,sub-models,,and,thus,constructing,system,surrogate,on,the,basis,of,elementary,emulators,could,achieve,better,emulation,performance.,Besides,,using,the,active,learning,one,could,optimise,the,designs,for,individual,sub-models,,and,thus,obtain,better,corresponding,GP,surrogates,,which,in,turn,produce,system,surrogate,with,higher,accuracy.,Layer,1,x,f1,w,Layer,2,f2,y,Figure,9:,An,illustrative,example,of,a,system,of,two,computer,models,f1,and,f2.,Note,this,is,only,for,illustration.,Linked,GP,in,Ming,and,Guillas,(2021),can,work,on,any,feed-forward,computer,systems.,(a),GP,of,f1,(b),GP,of,f2,(c),Linked,GP,of,f2,◦,f1,(d),GP,of,f2,◦,f1,Figure,10:,Linked,GP,(in,(c)),and,linked,GP,(in,(d)),emulators,of,a,feed-forward,system,(f2,◦f1),of,two,computer,models,f1,and,f2,connected,as,shown,in,Figure,9.,The,filled,circles,in,(d),are,training,points,for,conventional,GP,,while,in,(a),and,(b),the,filled,circles,represent,the,initial,design,of,the,active,learning,to,build,GP,surrogates,of,individual,computer,models,f1,and,f2,for,linked,GP,in,(c);,the,filled,triangles,in,(a),and,(b),are,training,points,created,by,the,active,learning;,the,solid,line,is,the,underlying,true,function;,the,dashed,line,is,the,mean,prediction;,the,shaded,area,represents,95%,prediction,interval.,443,444,445,446,The,toy,example,motivates,further,explorations,of,linked,GP,in,constructing,non-intrusive,ROM,for,fusion,systems,by,linking,non-intrusive,ROM,of,individual,sub-models.,For,example,,to,construct,the,ROM,of,the,two-layered,system,in,Figure,9,,one,could,first,build,GP-based,non-intrusive,ROM,(as,demonstrated,above),for,all,individual,sub-models,(f1,and,f2),and,then,3https://github.com/mingdeyu/DGP,17,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,construct,the,non-intrusive,ROM,of,the,whole,system,by,linking,the,non-intrusive,ROM,of,f1,to,that,of,f2,through,the,reduced,space,w,analytically.,One,key,benefit,of,this,approach,for,system-wise,reduced,order,modelling,is,that,one,only,needs,to,do,dimensionality,reduction,to,the,outputs,of,sub-models.,Whereas,,to,build,intrusive,ROM,,one,has,to,make,extra,challenging,efforts,to,reformulate,the,original,high-fidelity,model,f2,under,reduced,input,w,and,output,y.,Implementing,the,active,learning,for,linked,GP,surrogates,for,systems,of,computer,models,with,high-dimensional,outputs,is,also,challenging.,In,comparison,to,the,static,design,(in,which,the,training,input,data,of,one,sub-model,matches,the,training,output,data,produced,by,the,feeding,sub-models),,the,active,learning,(e.g.,,the,adaptive,design,introduced,in,Ming,and,Guillas,(2021)),could,lose,the,input/output,data,matching,,and,thus,further,explorations,are,required,to,examine,how,to,conduct,dimension,reductions,for,the,internal,sub-model,input/output,so,that,all,information,contained,in,the,training,data,of,linked,sub-models,are,utilised.,Recommendation:,Investigate,how,to,jointly,reduce,dimensionality,of,outputs,that,are,inputs,of,key,nuclear,fusion,models,,such,as,the,heat,from,the,anisotropic,heat,transfer,model,propa-,gated,to,the,wall,heat,transfer,model.,Emulation,with,high-dimensional,outputs,(GP-ROM),of,the,first,simluator,and,active,subspace,for,dimension,reduction,of,the,subsequent,inputs,of,the,following,simulator,should,be,used,in,synergy.,To,establish,such,a,combined,strategy,will,require,examining,carefully,how,to,weigh,variations,in,outputs,of,the,first,model,and,the,influence,of,inputs,for,the,second.,The,sampling,approach,of,4,needs,to,be,tailored,to,this,new,context,as,well.,It,is,necessary,to,carry,out,such,combination,of,methods,and,strategies,due,to,the,very,high,dimensions,,heavy,data,transfers,,and,extremely,costly,simulations.,References,Chang,,K.-L.,,Guillas,,S.,,et,al.,(2019).,Computer,model,calibration,with,large,non-stationary,spatial,outputs:,application,to,the,calibration,of,a,climate,model.,Journal,of,the,Royal,Statistical,Society,Series,C,,68(1):51–78.,Cohn,,D.,A.,(1996).,Neural,network,exploration,using,optimal,experiment,design.,Neural,networks,,9(6):1071–1083.,Dalbey,,K.,R.,(2013).,Efficient,and,Robust,Gradient,Enhanced,Kriging,Emulators.,Technical,Report,SAND2013–7022,,Sandia,National,Laboratories:,Albuquerque,,NM,,USA.,Damianou,,A.,and,Lawrence,,N.,D.,(2013).,Deep,Gaussian,processes.,In,Artificial,intelligence,and,statistics,,pages,207–215.,PMLR.,Deluzet,,F.,and,Narski,,J.,(2019).,A,two,field,iterated,asymptotic-preserving,method,for,highly,anisotropic,elliptic,equations.,Multiscale,Modeling,&,Simulation,,17(1):434–459.,Fukumizu,,K.,and,Leng,,C.,(2014).,Gradient-based,kernel,dimension,reduction,for,regression.,Journal,of,the,American,Statistical,Association,,109(505):359–370.,Gu,,M.,and,Berger,,J.,O.,(2016).,Parallel,partial,Gaussian,process,emulation,for,computer,models,with,massive,output.,The,Annals,of,Applied,Statistics,,10(3):1317–1347.,18,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,Gu,,M.,,Palomo,,J.,,and,Berger,,J.,O.,(2018a).,RobustGaSP:,robust,Gaussian,stochastic,process,emulation,in,R.,arXiv:1801.01874.,Gu,,M.,,Wang,,X.,,and,Berger,,J.,O.,(2018b).,Robust,Gaussian,stochastic,process,emulation.,The,Annals,of,Statistics,,46(6A):3038–3066.,Guillas,,S.,,Sarri,,A.,,Day,,S.,J.,,Liu,,X.,,Dias,,F.,,et,al.,(2018).,Functional,emulation,of,high,resolution,tsunami,modelling,over,Cascadia.,Annals,of,Applied,Statistics,,12(4):2023–2053.,Kashinath,,K.,,Mustafa,,M.,,Albert,,A.,,Wu,,J.,,Jiang,,C.,,Esmaeilzadeh,,S.,,Azizzadenesheli,,K.,,Wang,,R.,,Chattopadhyay,,A.,,Singh,,A.,,et,al.,(2021).,Physics-informed,machine,learning:,case,studies,for,weather,and,climate,modelling.,Philosophical,Transactions,of,the,Royal,Society,A,,379(2194):20200093.,Kyzyurova,,K.,N.,(2017).,On,uncertainty,quantification,for,systems,of,computer,models.,PhD,thesis,,Duke,University.,Liu,,H.,,Ong,,Y.-S.,,Shen,,X.,,and,Cai,,J.,(2020).,When,Gaussian,process,meets,big,data:,a,review,of,scalable,GPs.,IEEE,transactions,on,neural,networks,and,learning,systems,,31(11):4405–4423.,Liu,,X.,and,Guillas,,S.,(2017).,Dimension,reduction,for,Gaussian,process,emulation:,An,appli-,cation,to,the,influence,of,bathymetry,on,tsunami,heights.,SIAM/ASA,Journal,on,Uncertainty,Quantification,,5(1):787–812.,MacKay,,D.,J.,(1992).,Information-based,objective,functions,for,active,data,selection.,Neural,computation,,4(4):590–604.,Ming,,D.,and,Guillas,,S.,(2021).,Linked,Gaussian,process,emulation,for,systems,of,computer,models,using,Mat´ern,kernels,and,adaptive,design.,SIAM/ASA,Journal,on,Uncertainty,Quan-,tification,(in,press).,ArXiv,preprint,arXiv:1912.09468.,Ming,,D.,,Williamson,,D.,,and,Guillas,,S.,(2021).,Deep,gaussian,process,emulation,using,stochas-,tic,imputation.,arXiv:2107.01590.,Pedregosa,,F.,,Varoquaux,,G.,,Gramfort,,A.,,Michel,,V.,,Thirion,,B.,,Grisel,,O.,,Blondel,,M.,,Prettenhofer,,P.,,Weiss,,R.,,Dubourg,,V.,,Vanderplas,,J.,,Passos,,A.,,Cournapeau,,D.,,Brucher,,M.,,Perrot,,M.,,and,Duchesnay,,E.,(2011).,Scikit-learn:,Machine,learning,in,Python.,Journal,of,Machine,Learning,Research,,12:2825–2830.,Quarteroni,,A.,,Manzoni,,A.,,and,Negri,,F.,(2015).,Reduced,Basis,Methods,for,Partial,Differ-,ential,Equations:,An,Introduction,,volume,92.,Springer.,Rasmussen,,C.,E.,and,Williams,,C.,K.,(2006).,Gaussian,Processes,for,Machine,Learning.,The,MIT,Press,,Cambridge,,MA.,Rathgeber,,F.,,Ham,,D.,A.,,Mitchell,,L.,,Lange,,M.,,Luporini,,F.,,McRae,,A.,T.,,Bercea,,G.-T.,,Markall,,G.,R.,,and,Kelly,,P.,H.,(2016).,Firedrake:,automating,the,finite,element,method,by,composing,abstractions.,ACM,Transactions,on,Mathematical,Software,(TOMS),,43(3):1–27.,19,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,Roustant,,O.,,Ginsbourger,,D.,,and,Deville,,Y.,(2012).,DiceKriging,,DiceOptim:,two,R,packages,for,the,analysis,of,computer,experiments,by,kriging-based,metamodeling,and,optimization.,Journal,of,Statistical,Software,,51(i01).,Salimbeni,,H.,and,Deisenroth,,M.,(2017).,Doubly,stochastic,variational,inference,for,deep,Gaussian,processes.,In,Advances,in,Neural,Information,Processing,Systems,,pages,4588–4599.,Sauer,,A.,,Gramacy,,R.,B.,,and,Higdon,,D.,(2020).,Active,learning,for,deep,Gaussian,process,surrogates.,arXiv:2012.08015.,Stein,,M.,L.,(1999).,Interpolation,of,Spatial,Data:,Some,Theory,for,Kriging.,Springer,,New,York.,Tripathy,,R.,,Bilionis,,I.,,and,Gonzalez,,M.,(2016).,Gaussian,processes,with,built-in,dimen-,sionality,reduction:,applications,to,high-dimensional,uncertainty,propagation.,Journal,of,Computational,Physics,,321:191–223.,Vernon,,I.,,Jackson,,S.,E.,,and,Cumming,,J.,A.,(2019).,Known,boundary,emulation,of,complex,computer,models.,SIAM/ASA,Journal,on,Uncertainty,Quantification,,7(3):838–876.,Watson-Parris,,D.,(2021).,Machine,learning,for,weather,and,climate,are,worlds,apart.,Philo-,sophical,Transactions,of,the,Royal,Society,A,,379(2194):20200098.,20

:pdfembed:`src:_static/CD-EXCALIBUR-FMS0044-M2.5.2_SelectionModelOrderReduction.pdf, height:1600, width:1100, align:middle`