From fc904af1fbbbc4b03f29f85454f4a2de5e9d6c66 Mon Sep 17 00:00:00 2001 From: Manos Date: Thu, 14 Oct 2021 06:12:28 +0200 Subject: [PATCH 01/23] evaluation: Reviewed and replied to Katerina --- graphics/evaluation/dist-cor-mod.pdf | Bin 16040 -> 16569 bytes graphics/evaluation/dist-cor-wk.pdf | Bin 16037 -> 16567 bytes rslt/bgt_cmp/Copenhagen-sel-cmp.pdf | Bin 14997 -> 15002 bytes rslt/bgt_cmp/Copenhagen.pdf | Bin 14649 -> 14620 bytes rslt/dist_cor/dist-cor-mod.pdf | Bin 16040 -> 16569 bytes rslt/dist_cor/dist-cor-stg.pdf | Bin 16037 -> 16568 bytes rslt/dist_cor/dist-cor-wk.pdf | Bin 16037 -> 16567 bytes text/evaluation/details.tex | 110 +++++++++++++++---------- text/evaluation/main.tex | 3 +- text/evaluation/summary.tex | 9 ++- text/evaluation/theotherthing.tex | 75 +++++++++-------- text/evaluation/thething.tex | 115 +++++++++++++++++---------- 12 files changed, 187 insertions(+), 125 deletions(-) diff --git a/graphics/evaluation/dist-cor-mod.pdf b/graphics/evaluation/dist-cor-mod.pdf index 4f652011ac806c9ebc7faa6a580895b940d0f26f..0d15c3df3f689698b63aa5ef1ea62e29ed16a3a9 100644 GIT binary patch delta 4409 zcmZvdc_5VQ7sneyO(I*_%1{_hn0aSMl%)@7%&67G3ctue0?al52uo!0btg=_cj!JO+^r2q)4 zG;jyPW)>LjTkh+7YN&qHR=pZL_6*t0pI>vY|Mcp|_mW@ySse-5CP{g$wzK%$zAHl? zr{*&v$Gjf`D64Qj*qwiNqt&wsBUj>iZO$_rJ-Tf|XKHQ1YpVfA)})kzmT>pmg62@y z(O_35BRKc9?`BzfrbK`3x)igYA8Wjk_q-LLex^fY%1m5wKeKF)vGQpQ#ZYis{hD1< z_F;)*PS?Z|L9?R@`T@Kn>0iy}hu^~cqZm6o&a2$*Di zy~Qic?2cesM8;G9T*Zq78Hzb1p~ksBHqn^8$1S&>dTyJq9P?!Ah4Pa{t~#`J-8IvH z{1G+Qb1)~18GrHcY7J_4ND;cnW zC`gnD-mqPyV0E5z^7hzhwUB9-ie~qazh24$G4A5qMW(|qzmJ+(|7PG3LrF0&@=vxB z+^D{VWM$2T5y6{ZUPs()g*E7Pd2D)X*eog?#cR?LEv4U3>{Gsuj+3VYe?>YnJ zZ2n+Hmdp1Ef4CvzP`egLEZM421saS6n%=3lOgAtU=ZQaQwhXBc)m2QBz65XC+FPn2 z8dU9QAe@Joh&Gj1_fuJ~swK}fh8~Cel+v_8y&E&VUH7X@dZ-D*$LEHV7^NNdo%6k& z^itpBa*}dTE@8dr@npo-jES)X=~ek7Ow@VhO259;cWn`dUhHrXqrrvbI0KcmNNlh{ z+6nrRj*yuAc;oZk*NXH?)s1fI6+lGkFml#?S5{`G>RVREFuxT?ZF8(>kGA2Q(1%SA zyDk7igYzDD3y;Io0pzohmyV-IHH@ zE~{zD!`00mHrL49^fa-jL189zlA$XmtBV-cMQHan_!zu#8lO1JXPT8U9l2ZANsE-U zYHd=WXHwv;v4OP{%!tgcNP>Cp^(&|JwZ;cl*F9R@AD_k#!h#Z*dJC8N(klxssz$Q1 z<(pDm-78$2PsU&jocyc?7;#D2qnZk518#yo6B!Qnse>H-Qi1u}$9l$wwassyvq>7( z!6Fvu{VpiKVzQ!cTobbG?xZ_t%Osr%Q$vkh#Jd(vbd`_b%cHz=s~ z#(q;f@A?8PjVHEFdWX}*s0A(Dm{LY@Z%zTx`q0ri10~908SMgQZZueEqN1s*R+P7dcdZA!vJ$mT)@kNyb+Kr zR;)!&ucJ!J6Fhf6PyG=2#GYww88(@rd4M@3E}-X!^tLJFzP#OL;G}wJcUCkzr7>+2 z-N8j_BDK*Y)l*vf`-Q|O=_Xc!h@6j#_~|W=&iU4DBg}uuBOAEy*e7w_)N;RVO2e~q zpBn=@V7(EdxA1a@3g)$4{{EuSsLg|(`}+C3`dO?7B0F|6&uMkCHa7cWM>gh_U7W5) zqu+g0JL-&^0hrY7c z#dM8yvk|>n3SO#LSCq+aSj=Zieuo6DbVfUa&e(nOwA(pSH1VeErJt9h{)PgBIOqo5 zGY>7-#pUlUdNo`W+~EOL9hCptDYE+gGx?OrHZR}t%g4M&MfQE23T&qd6b{uFMwz8N z@DfsC2WhjZg{Na(1ryp7>$rhq!{V$P*;~kV)}y#BP29kL)Qo6aUqGi(zQurn*~5&X z+%AF0A*K8wBYDt7j<%(#Z*(G7)m)=}IMaibP4zxcfns-etdMuJ<;)a3~c$74gn z9U3s3X3R$7kX>1*1~W`Um28&~WMuoR^vPVC263CiqkI)!d9O|v#dLVMc^RE>{%ThS zE(UbY_8Sr&cp(;hWoM=Lo^>qw2V6GSP;7D}Fm%vS88kvroD*gQ`EZ!4`;pc;n!cURyIdV1z*`G4(! z(M%MN>f#wtH<8Tg8eV5D;Ig5wG5h+}$%v_uve2!}ux+YS21?&AC&CDN9bNbM5)=*YnaHS^IcK^dcm7QVl!e$m(2IiD2Pt{G?jwAL$0L1FB?hu_=z zKf#r2s&{kIZITC%`oC){`82tyU}(kg#FVP7w{_(;hE8)L?yC8C)1l9Q+G`C8aXMZ; z6s>5Jn!o%pFRI{tsz{HYLKwc}NX@uHi%duTz(l{hS=_hE?$J@kk2$8_IijTb(}5o) zS>A2ybbRkePE|T@q$r#Wc>DC@-H>Vb3N76);M>Ds$&9aiwNTZjH!}QEqWVEQEg2mT zjy48;3(~S<9FrH79Qz{7_OBTT8ljKsIXHBUMd$7otT67|EZccI?X}qfcY;OPj4{zJ zIG43Y46HEz=x%mpJi|PHR8Oyi!+4OsfJi(esg9)4kGr#3G^f+N8ROU=NF0E|q9E)V zK|U&pLxF!lfP_cs>LM52yl)cszd2ow73TUOHTIXywtu_h-Jdq+V4~KqD5hvKe|OOF z#TthVG&vv9QW}xq2@1Ih>=oa7K}SejsJ5yjYS>unZ1|wmq=5)ujSQ>p`d{0icXh_P zgCg-c7$@VPaHxuTq2qmrq?DUOR&~8gVmX-ftpxc|Fh@Tof)IzM`X3i>DYPsU(?mLL zH;cQOEVLK>qW@zesvjtnXgam={gs;SN4vPivw9fn*rxPo+jjEHE=aF9^?bd;M(W1L zRRvDNo4@X?Don<77Vr9M7FC_H&N_Ft(V7!Id5NMJ=iOYaYf|VVV36QG@YQj%iM`7P zt{>mU$62JU+YNiGFh6z=98S_pE_Tx5bgnHB2LUSSH#kNI_FU%Qxr#f5nclIb$)*cB zKnSZ@c<5{y@p=FAt!2`dL_^vgoue7CX$QB>kruvTZR$Xh`7vE))tRFaA?Mhz1 zFSTr^D9io!`>JDi{Y(7qpIYlbPW`JB-NFdB9mBS_Djej$8jalbR;Bw+fbRVKN*9DS z!R5(ru1$pcc$tpDp?#aGl2MfX215qs<2FgoEzd=T^W%juoowtr3G<_T@45V;JJ%ga z7CnL_r9~fohK^7@Vq0p$`q-=>s(oe@+Nro5ibIt^6eTp|ugHaPThus?O7#dxRv8W1 zY$=0URMa>I%2yGP1sa{JYJq?rpw%EhG!6<Su5RUvD zMB&Nsv9~~JGM!j7`YD6mizXV&nMP-$2-v^ZEEEC9n;|vbQ3T%o zffKpAd7}vU#W~Z-nT8?|{$9ax#KoB_iw4_CDALamwig3M_*n;1(-TEl)b~OW7G-@< z#6>}08jFD<;y8&E5d;E&vZ#uya0DzKj{}HM6V(@ihYo0If<%CKnRx?6ME#LfV@0J;!i5@?EkxgB~nO#SNJa*3C}xbyz$GA^dGS#GWjNX7%CWqzdpNJynDqd)=3OScJ2AuQs@g1id|;I;ee&Ia5&h J*~oks@_&$G7o`9I delta 3937 zcmZuxc|4T+7rw47t}Rs5l_4~Qd1s$NDqAR9a#gsN3|Sf(!&qW&6xmX}_CZ&-v|I@z z#aNO^M4>3ln8;Pe775vY@04yo-0xrC^L)-Z&v~Bnec!S(d^f`jpYhRqOg2|c*-uD6 z)PK<1r}#wdvfPHxg=QtM4U2^hFDno)Cz{5~&CNJ8XN*ORi+gIyA5c*7X{>Xb9^ks+ zV?36CjJd`9A<5Y}J;t1P1lwxsm9adjVhti^ynE^XeajD*e}-A?Sq z9ei{1kLB~nQVLDwQz-MN>D_z6M(t2PNh~adw9W^3p!B^+%_9}I>Mb*!Q`JLBQ(V7M ziY3YSWBs-FGD7aibnmFz_QO}r*&pQou}ALrNzs!3oPfN|Nz;<8zWxBc{g}F}@aYCX zE|1o3HMQJVYWCZ?5l72MG)oP^fv8PY!sKkb)=kXADer;P=z$K;b9C(*vvj|0*`fs6 zWDmaPjzr@oC&&EYG}j4_((LI*le^i2RoS~5b>nk}szk_~>$4Ya`WJLg`5&Hm%=&%}b! z-P~Je%s!M%KCts_eJc{pH^DTS0~9iLYETY;Tk#ha1N$BCKJfsJf= zkBmp{u`l;#1}_YP;zzGNeHfCNWt!2PaQB9g+V!XAL$by1PewXyv)f=cuW~55zgynf zLR=2ffMUc2J(_)J;}qQ7LC8Ug^*x&n^@dp}mt#B*TR~lldZLarzKMnCAteF)A^rRb z?>NRuGV~sStX?0@Ek-TY&_cLnXY`7MObYX4c3VztRQOwI3Y*&Q+AE;s&6}!cZ_e3k z7oW7mjj0iygEB;B#>eTNhb51tYon#YrP?W+4k35voAZY+p#}O{X1&QS$A%-FC~zAp zzDAskRT3Jid?p|1=W%*B-bHUmvtu%&pN~O5FCFG2`A*&NW3w+Lu}#3!RNZ6;v@xOi zrzZgUcvJZ}UIq4|ns)z45dkvtt6f(2eTNi1(e&40b|GeZF>fTg?^d&xxc^Lk8N##X zCeLVD`yOm`)AVingHPi|pt*664F4%E<#T`PK;$2+ zNTlSHkv@u<>$^pY(KQ2U3r9q=5PDCyCe-fZYHZcUhCb4?>SDKh<4B@|QL`7WE(%Z+ z?G=qP_yd>p7GwKDBd!m1ZN7QSvgKt*!NL4nh5LV*8tg4$r8_VG;qqy!jy34jSNG}@ zcfp`KYZ>jlKnSNJ7csdvKheM77_wV&B3&n?N3xn7z2}A>=j(FzKPY*E+s_JQ2q@!( zb$lg85-hgyk0mw4W(_ZKk@}05tlE~Ecb<#3iVV6UpgYhW64mYcX*%TRypH3i>vvpn z*TG$p-KsCr6)ruJVCDCsRyFSn(5Bbupx&iXK9oLhp zM`SLUB(`lgC?WRzo^Z#kKG&Cu;nMU3*GU45n*O_S7ax(IP^PeN= zYvJ>l{$f?e$9m5^ zs>nj;`CqkvYJIIBk`({&1ZDI4sLQrh?hy)(r4Ek5i<*LnqjMau8p#G*QMU21nwa`A z({$OaHcQzG7jd~yOw1CKI(vjzT8?=qBB-bw6DgC}duakr`0LJ=9=ge2C8f{X-hL*E zjtzV3xF4E`*30D>zOz1cwMbWbyP#*ubR;WN=g1ZP{WZM-vI`^H9RI;F=@@UxbJ>R# z?&V#Rxo*OCX`Rw^?caZkYcL7`3o^S?^A7wb#8>pz#$&?F<(KI{j{*yv61-u+-h&{D zk@sh*(UbbR5Jr*WhpPhgfilZ7i{CD@$otCAUfl6&H2sBYPj1swi%yqaqeU(vMl=`4 zGzzN#Msv#2J_6qIjXFE*B4J}P+9Y#h*dEK?*m@?`SoCes(S2j()4Kd4&&7VUz{8Tv zJuD6Y1qaHU>@PCNCFzN0u-Y{bvt3ifAN%50UB5oVft?dfb-rjr}h5 zWj;;h`{;FInq8A@g_YWqT7P57x#ytt*|<~+qdD-9_8Js*_iq;G|j~z@qCEetWssk}&X!}d_`kRA?-+^v2l8n}CY@aW5_bVI^4nG*TjnkX zk4tdE2LYi_9!(y4F``@Z~e@Zjf+3qO;ru4_R62H zvkx3PvAfN*^N#Z(4$na7-Ci16YWNrMe^yr{BR$a1(>5e0u z)XK-x+aHYnlsY>vu0_0kylG3e&)w9rOQ)U+dow6vn%GQ z^e0s}Gna%(IZ}C|B(i$|l|pho$-DWvd5a=ZFaXB8pwI-oDAV8K!ZFS?Q# zoQ1v6Bwh=M8II>f?r8Wz%aJvf{IJw%ZcafeiZM=)@flDQsy5}s&itO3xve_BwB~m) z|1c#Qb z>7dDvyJ=Lb{jCL?j!+sj1=Eb%NlJ~C+)%L8-)1?DUA>L9I3?`;3UdoAl9OTjNnDo( z&nZ`fPRbe}c}E#&xI>+3t%R^wvOu z1{&&Mokl*Gr2%JV?DFFS{dd8cfR-*FxIqgJg4%S@M;i|6Ad10_2n}#E&<&0v;7oaB z6Q3XffFZ2G4=6n*96c|{XAPJD2d_+!vai(jszbc61=Uev;iO@(I_+u1Geji@}WRW zJp>(t2Ovt`fstXNUoX&e0H7XgG2YN?FdR?PS`7Cec?3x0syrybL6QJ)K~IGb0F`#D z(Xs0|(E!hcRdq-dfoIqn42^<(SdC!-BCkJm{?!Gbh)}c!L!*$8bI_sxUm_aKGm&?G z6r-`YwIl!rz^^qAz@SmYw4v$~w0*;8JuTz2t@aq)gk;HY*;IZo);tAMwE)b9y=;5v&nt=Yk68aCt1T1>( z8v+RUb?*ZpqQ1T%sN;`XB8IqbXd;pKS*;>Q0?^`DVUYhw0E1i&@-Q4~)t;a>NT_;U z9uiGhs}708V%HHP@%Xi?L!zK*WLHfCkznvRH0ZVW*Ka=y7~+th9~B1C@_rV!UZEry c7WM6~?Mk5rQ%D}7NDL7Lh^nd@TbYRd51Qjwxc~qF diff --git a/graphics/evaluation/dist-cor-wk.pdf b/graphics/evaluation/dist-cor-wk.pdf index e098a935ff3b5599bd8b21b568a2c1a5b19bfde1..4b418ed5d2257a34ec3219927e8d555e8ce6dadd 100644 GIT binary patch delta 4379 zcmZ`)c_37K8*V7NlMqQ$WC)Eg=FC2pN+hx-OBmT2BFoH}vLxqLZVC~H#ukc{Qqp`S zhU{x06(u_%ThW3m>YFKjx47T$ulM(!=Y5~&ea`Rv&b5;Tr(ky*VPLIPadTdx@rscQ zo6@iEGNp}HZiETc;+c2-k%s%OnCt@(hLfeuW9<{6a#~#FzeL2YKCDho9wp@+E%Fc^ zijjOG=dJ}-fleNYJ$qqWsT1_>SIJvD!ogG_ z>e>~QzoG=0OWcE5KynYcXf+OPnh0kp~+nb>Ce{9y4N_Ogt+*X8aa~_aisj+O}caL zSI`f4zUtXdo<8{DHozs-*m+gW4C@lhwfW?P~skn(H=^RUC;#+Z#6v z5opT+t2%Fw5CVa2V;^qd6_(|$)MxBfwZ786ffa%q7|TCZN0!0#@1g@T7ihL_0GRpi)i|ASpD#$YUb%=Z*=*YyHEN?^nIPvR?e#*mX2(Yte~Z( z)+IGY}RhFZpnS{%zm)_8l%F&_uza7cEPGigs-{q zRM0bn*PT5H)yhcrQ%X7je9|--?`qv6gw~ddIranI@ zQ)B#Tu&@4RDZ(s2e-oz1l_FzchPkYo+lq_VMCY1=t!izSR*#jvEC-b!?lIfTL%g;9} z5lznytGlN8wKw)OW5CW61e%d{3`f?dly`0$*&Vl1z~YWZ+Zb0yzhzB=N8Dy1nT)Bn zA!_RaM&VsZ`Q1yTRB;nSXZB`ob5@MQ!_ml{g{YKJYE7L1{l+IX^;_F4=3tTyb>WsX zwPM#UeiUxbKgS>+ajd>-9H|3)?EgTlIN32tZxH!t;4K+e8FE5=mGJ2NvAMP2>%T96 zmmM6OO?{@H+H(QZDpS!-B}M7Sr9XC*R*YAKOoaxu1u%3Lv`g>h4TFu+J?3CxVrb&s z8}oC=Q)CoUL}ZDG<4x{4w?lkfU9+^SuW3Cnnjm^(&$TPYdrKUzQGO@xfIpbZ@cH<8 zeR^Nrwxi$PL3(eXE0xfEMx|fKyPzNx|dG1hvtX+Y`2+u`49C^`ydyO5TUXy zcT@`Y#!f1x4AvxIpOI$)Mt|a%dixG4e${IszQc4#=KYWIkMB;ufH2BCY%MliFQj zrI?i43VCNVd?Ux4ow9oGCyEPd%!05Wv+{654YKEWbeLT^WkSE4G0=bYWNZx>D|$V_ zEzfvw+J3qOWqm3ikkfnm!67=k?BLj;tjaFq^_>N+EVbNT*&u^*qc;}x?0hYW?9*#r z2bZ5TmZhdY(l1{yUOzvM?)4npA`$KdRhZqfyiFO=&jI}CEguzJByPL_eXbnexbZ7n z0By?7je<*54m{a-uVTZfE~kGovjAoYA+OXYIUd-5;_5(6wXiLHCsrS!?0?yxo2x9= z6)_^=W<(8e-p6Jb*~W3vy*8S?r7^Jk+y^4fm}Obgx-}#8#_;vbjR-+%O}@CfTA0ZB~bEiQP2XWwEjERN#SM zpK??V%5-xDEpJvS?$?!7)zQ9sdfc(%MQ{uwJXb-?u~WL=%<9w6>My(h@Id}Y)TQJx zrjXj-dFNKW=;~?B4UK^uu|vFGFPA>+O~fUC?j2<@N2+6Acj62htc*tw#NAO0=$h`<7ii&% zKt-uqEUCgV&B%@g)c+yJ-(nCVc(N^N)VPE<`FRW>b{?PmI4SCTZNG>KaE|x<#bD=^ zTXl{ksm{xP@{Ia1_Pe6Mqth)7i`qczH+LN9UT4@P#*1*hf@E$zST3#ZH|eVKXnYO2j?fJ;Z1uBP(?DP_91;GC+a73 z=%f|YvG)NF#&4Wb?+Lt9;&QWR*e4*VGXG&~%w6p&zr5-qsd=;Bz^zlV!6f4kStJQ#J<8v@9J@-i_AbV}QF99UZ@ zTAPlu0X3VG^1lXu{5)hYYE^tmPtaOJ7dZ2DOJ^?Cs$y_giH{f%E!;3QEE3=3ADqGM z_|p869l*@$Pk?R83EksHW|7+guV{k?I>j=QopFb18NJgjYh3&K-nr-3J7&`5-c^%q zo_||GLrZeuG~dJiEGEsxhadU5dcgrK@(&D>A7OD64t|ZW0Ei{xkUMt37jc291dGQ$ z)ff@(2{_uKY_8*DR^Z!fU!06p-pa3&H=e&S>Rh?TNt(7fNbC`ffIkijd#>6iu{}at zSVFkDq4VN|p;YL(QK@M?QGq4|^L5e(HK^sO;f_(!#C%m3!&B#=26{wiYv+0?Pp8|B z&)n0h!Ss(M@IImZ-ErsfY!oHru*9pv1BL4};4V65?9>e5eW=Q}eQC(I*h0xRZ`szE zCLL-ww|H)+DrIU*cC14Osj3^=S)6(HnVc*|_I|@dmx-<8x($UHs$Inf<7O8duZr8= zn|oo)iJiVcmS+dPEZ$*U7_>?+*?(l*d8_ducWLekfyn!}iQ9Gb2O3nr?j1RpzB8lP zMH6a~l;ljTy(R%-Da79l+Oyv%T2Oa2cSdz~mtvcJH?$ug-t;Zc?ZEjVi;C@K8xIil zX<6FT>+#pD)c($od|DeHkSP^cIGC?QMWnGaMawpfnuusWp7N1eN!rz*|7c%>){*OD zQAHXTzh$nRfZ=E&C{OFbNYjm`Eb!@y%b-GBn?3&gsdxmtQ~pJ)f9U$bjH+=Zv#6#h zgWk(;la(H{m*lqy_%Rx@l(z4%b*NlGq}%4Y*%)MOFv%>_6=V8eX>D-55mFLzsML1% z{mc(ts8@dH9HuZG?Q&Lv39)d4C(1opCwI)xuXKlL;oTqX<=UTD|5T=}XKk{jAp=Ra z&>Pb;|74%;_Nqcmq##jPwTp!@ku;|Yv~mT*bj9s5#O&b2Yq8XF)cI|84v&nPeJ|z2 zv>5N99r8L*JhB8LD4?Jqc`k%jRN*)(JcB{fN+`%yu?%WeR^jYYN`OJ;C=_Qw*%Aij zpj4nh6aa;)pr97iZAeNL#mQFjgF%66D2|S*77UV8M?o2CAT+100vW33LknmXNCDjm z4 zj<6UVVbXYcB9i!>n-$=P#DCWT*Eo*EFY*T<@r$%UNWvmvFpcSlBmkTgvM3COg>F*h zR|9wq4hIkb=s6`AhJ(yBH9!Iu;NkfT>5GJa4*&uNATH1Gxt8DnzLKRm;=ka?Jaxa} zcpD;bKnygmxfMo+kXqY7;xbMgfp6JwbpQ^>H~9A)mW*A7!xJbxlYhfuuoTQvy%-!8 z%h!Ryfd3f4Gl{TN31)D}@r(9jz<(N&Nt9(I6wFWHKR61Ou*@wAiLh*93Wc(i7{K6{ILPxK zz>tBZlK>bBW?2pZfTeB$0AZOx9%`8j02Z_SqrqZ{#K0xC0$2ie=^^BCkbbe1#dPuX zrZM5#+DHu}Uw7WuzId-#`uVYtypj2r)dA078j|PJ_rJ6YlNHFMxx)bh9!r8NDeW;g Gfd3ba{TCSk delta 3858 zcmZuxc_36>8*eI+gx8YF(%6?Yb7!AnL}<|>qIk=` ztEnvIElFfe$o|PviEk#=SG?!1bAIPM&u=;B-hv=lTEM*q7)aPqI+Hs*-@k-}xYjhBa?MAMAzYM)eea+4K1xt}6bUF-!Z2@Er{i7A)pVYfJaQ=sDd z$Q*1TAldnB$U?`y4M*OYp*C1iYAd|BomGA=9+5(a=6hz`*o3nA4V`m=a|A1x#(+WIdO^#Ev%FQg=;O}&K=WP+1bd7 zBwjLncBjTjAY>9L*xW6$TamW5#Xv%F+(Y~8(STFpR?;yf@-?CLNmmfEiCe9k+eKtD z=~~AgV#*UX6Sv+qQpOot=3gFH#Tnji&AnyHC1A=GWKWP~u$C8W-!+WK$UfR7+f8&` zQyxDt)m}*suTt?)1cRPQR>KxT6Ws6W9N)OO<2HDAZQ((g=3Ws_Am)`(j(NXGO#zQY zw(zg}PYaJ;IH1?)9i#4{IEjmqbkt0wjKj5kSUST^jE_X*eps`ai_Jn5fM#J>b5`B@;TT#s@%T?^ZY8PK zNme|~55=nHCU3Zebv4j(GuEV=oEu23y`huTeNslkQUft?#d+5Caf{Wem$k6XnyOi% z$B6=5_geppeg1C~qh9~aK!I74d){HE zd)s&TB@LbUI>iKAz;}lo7v@|EN4B492&c3X?+l03r-%_^UT67vRQsS&i*oW0Ng$tzx!<^|l0DAhd0^tTcw9VxuDGomK(^#SeT zzu!pSYd?PGc~PEA6oP)BRoDlD(Q3VeAN7DXs)_EX#sO&ju-%z&>tANay;yJXAs8e3q+foCQf zypTm7A;WUPx!k+@7PXDGb=AIjgl%wSn&~LEjFWX$H8-w;i9q(;~9iuc@-} zx#CsFY#XIgQ9n(UnDg#4D;*nUwy2Fi<|G|u(2vx|(9V$4i-n}EX4@3w^Mp$6&NbC; zPT@?7fcFT6@&rq8J`3RjrAFjY)tcD(qceNRx;Yup%-8-nz_FywSSt?NOY z_$uTq4D{Pu#o2=-x(!E$9;zaZ7*|ca=}8HQc?`ygW?Xa0rR+|&@RlG+CGihrb_G9l z^p34OIFy`L^Ug=Az2Kd%c6OJ%pJA2BdkgRM`??b8!Ry|#s{$zU?bo;OuKMO9H8-r; zMekFP;J8E8=2@0^NrT3jC@=4qliJP_w_lwEQ%`T)IKLNGkWR7LB@o_CcNID0yl(2o zA^L1n+lpHzQ7_)t=Vad2Yntfl3E9e0*pwmy6*U$)*gzc5N50|rRdRHJw7SAuTc0|} zjBChy*GR-8oGoL0I5|pU4s(SAH@sU#H4hxUeEUY^NH}4{BcUr^!>L6m3gLjHDK^*n1#;+A!|NlLH|zM=4q zAb-<+sD9BBfIrk!B4Sk>K`Fl6eu}=vX6V?MgxV<*b3lvHS!Z{Fa)RHwD7CLWC+n$^ z4AYi{|4>#3{_?6vh&K3ys98*6w7Bshcfgn@?IUCK#ew%XON{>-a|m8^&T+-%?Ba9X ziJ;_NCO6_5yQUBDN?#3z(RuMR9g>1H*YNhC;v`*P>7t?Z;0(#AGD_U5zV@40PaI@o zUANr*nK`&~v{)bm&uT3RHF0Q^FFsH9&0349>rnlf$z6(6c!zwHHML$*E7K@{qii0p z^omyRx4kYP5VX&u^WUo=oTJB(#r%r1q1Epz6HgUc{tU{T=Ni%`N(Cp z^)Oh-{CFf1c?X7#V-T{zS^Co2~0)GFukP zDvI9Yti?)OCr;gE*GBn?#ny>&qvw#^ z=G*M)@J*qoD_r|hBoKlhPnIz0WQ$0B>J46F!Qzsluf9{$Hj&n3O#|iT=Ts4nXg^V{ zzG=@U#BTQqiC0%C&nDC(t^_m=M$ZrRHk9m}VfI|OGYD3cil;;Hb@DJ`{`Pf(D?t<% z2j9M3*vFSabv`M~O2S3ft*-$@jK-3Km7jcJCQI!U8=P<1yO)95B9iDo3NvKLmrEyy z&p2L8kgd4zu*y+{9<*u48*lB@)yEBn!t@mXg1B#;jrzq8m@gQMB~f}5o8>;@#S+fS zjLh5pZ5tWm)K5T;(4aq^6<)?NO_Q;A$HAfO-rJ5>nU9 zAJfgxnAs5D}RN zi6ONhDWDA+LTcp8pz2@(L;#Mofcnwod^OBh0hk4(M5+dz8B|Xi)zue{A;1B+B^>F^ zM;LC$Pw*h(4~WWep?dPxq{TG@js+HHK5#6GKNCj!xUt|^^bgeA*@X(nVis4va4h!6 z>;!|#!*M@WOfNWA8^5?X4aa`x7e+GS*dMfhaO@94HkIK8#{v2AI-)Qb3c9VUCGzntUoawAP z0RZ6t0{}1>>{2lR28Ult0$>5c((nK*2EB|JOIRKNj&~Wq3*5{Xfx}`J)%+#~M}X+t zB|$W>)Hi_tm(zIsvdjQH0l&;Qo`hN^od6J)StOuH-=hG3dmsP-x4a>Nh+F2Hh~mFz z{`mE{03v$%CGbj@J{o|?Bl)eC$IlxiG=3Qg2}@e0nndD%Z2a+y7zJQZOG!|?8(faz z(SKYFuL%kTfXfu3P#EG;cPJDNw~QD?z%RWz6dFJ;!>|PWZ?(P*XEz#^AxtL2ktUw5 x1m3s&{jUK>T6%f;!hg)I+}Kn&FZl1jZ)b)ti$Qf2Mqx=Pg0QOU&V9Rt{|6q4GNu3k diff --git a/rslt/bgt_cmp/Copenhagen-sel-cmp.pdf b/rslt/bgt_cmp/Copenhagen-sel-cmp.pdf index 942c6b0343cc6c9965fbfc90d682cb7312ac2d23..3489247e9ad0ce9b75619189506f2f0084533203 100644 GIT binary patch delta 2650 zcmZuvcUY6>7cG^F1Y`#zpoCS1jC@JHF9{>;5s{WHpdeELfds=wi7c^HWCR3}lr2Lb zvWy^tiZW#iXmGL=1PU?*v=;mP`uij4``3NWz31F}&huu4=7!!L=b@=Lwx$Ls1;evs zEb@AI&QX?)1kdh)(%SNs`>IBA*{(~uxiSS|`K4oatnG>ux3)JKAK$Lm(NbyOHY)Wy z+rk--SvUIACB`d6G+Ki`A2)XCn$@uGZu{~qlzgxL11dOG^#eor5y|{cxUKuR)fp=h zLkI765rvdxnDupsv;f*BRUbIhB2f|**sxsFc4CvZ!TQRk_S_OM9a5QLe0qWOp#Ekp z5bnD6NM%aRurv=jTqjm_xlS?e(=yLHbiJWr$msor?1D;N{xd&PZzdl_%*#x_ylE74B zCo7IC7GKeP=;Oafiv;a0w93q?vT;+Hw^BMG3hP2jhje-Oj_h?yQy(^ZVW8f2OtG-o zbIUtomdDWR}$g%S}Q6hp=M{hAd zr-}%@whNb#RA}Oj3>a_x0Y#m@=(PQgIf$G9hXm}fdRKYK5fK@;v<2)^A@ApRAi%D) zI9JFbmOIKKnvtV^V+xHFVsZVk2DJ>5U%q`~I-ighDJ)}Mg*Q`7&Fq>RoHP{Xk|CU; zi(Q`T(uOnE9{P@^%}{W?=Y3;G$Fli z)1tEz{B?1=Z=1F1@a=soq@X&JZpsrbvgCzmnQMOAmda<>cV9Q-w6tvF*&Wrr`jd@u5 zIe2q~H2j@tj?S%hL{V}anr;YCBQD3LIH9Jam25ERX#_$qiN=@hZ_=sCsy1uD5yL?b zp%46y^4deg?neo#4ngW`f)`FoNWou9Cz_4e&Il|0G&wUEnh22BOaonBo-d5rMlixOdQj|G?$8$SDk&bbQy_WM*p9gJ65wbW>(Jq=#5 zpyu?#VYhYppr&APs^^E(3xZ8HQkG4}(|c;-5?vF8ZexxbAG)BM?3zX$i0h(L~U(Njm&_3D++OEO{Fm-VQHq1rwU3>X_A>inoGo2o6P(#%iF4~ zs7ZKz^`={C@ei1~ww|n&J+^o4T5>9TqFjA^tfW9poOJviW0kP3iabqFAi-`{7LZ?u zcO*_d@pC2@8kq8%1*Q-^l_RU9)QDbTjyg|i)Hcw#%pAdHX0%Jpjf@@MIP^8Y-+1n- zS#9W>%Z&Fep5@xiapy(UTYReU>v`Rgs)RS&ynd&~BBxsrQbJ={DKALO6mgrU%!^r>5d$=4nAC8Kko5|4)b91K&18zw2()k z0h;qIe6Z(UuE(EB3^*LQ)fD(Aj!1D?D7fDXs&5mv9_Ru0s_C(3y6ycZrV13qiYkWw z8kVzctI;Cw^tLVO2ermHZ_eMlmm0nreWSMFZpW4VZ#H2&YtxD@YT@p&>#X&~Y`zpx z31X=45Zkz;^$m0BZuKI0rQNRNSAEnB?W_NSo}P9Y4#U!`Q@AsMXaJJp2VQ$_H3d8u zj(B!T%EqrdxMo23$hh#6)30L5QZjw9Zzf*}b@CmainZS|Xg#}PHRawv#$I~-R{tbz z@|V1pvMyI<06j2w?MBXZa{c5Yr16oh(Z5Lfk{LJ_x#O;7&zk%HVi4P!zpTw{zibvA{!<^G$fpVO3G$|fob&Mr z{9chg{b2wai9-HU;IXKBasDB7J}e52#e;bIJXq}mB2nKTG6)I#rvNY@0B~bi3>OAr zca=G~5cD5&cr5)Zq{9Qz2{>gQknVvK*W=m>;Bj0`0E6ekFaR2&*X}An9J^6y5X5s~ z5T3w=5wHNA2QRLN>0*;U}01_6*WI}`|^IbEPYJeEK&+*RPr z0R;gBE*B66;!GL^VRuvggcqlADc}Gsiqjbmi{ZKp2jaQ81Q5%~#RJ@Pcr5TgPj3JJ zI4C>;;0lEBAAt}+0$20|0*><-0EL1;`UF8-k24Se1z|WN1yBSur$qoj;qi1pTYUG^ SnQEK!qtP%WCDUKcV1EN`?=_+T delta 2784 zcmZuvc|6qlAC3q^yOgsWqsAmf-|yVxY#ce(Dni&IS5(T;7)JEnCJa)plp|M*kc<#5 zY8=TC!%*W2Q^0ikn_4zEX+u z!WN@jh=f4p@W*_+n}*uES_@z8n$f7r4Zn73aLls(c*Tl;#qE^o0M9w}Msa_xvCT`87N!n4m>~|Vw z(!69{vL({DjM~V_O+!~EJ>Ey((YIvWkRDtNm|A13mFrzD2(J*eVvAEfv~hXGzZ>R- zS7`(}H$BmhZ4${TJ18Kk&k-&xm+*9KB6XbfZ~vka5r3jCx#py4T<9Yfm@8-57s1_s zE`eKJMw3B2h?Y@4rs|e>sm`eRoujyWqA(bkeA4c=?L5s5dlNdO92HiGL)`1kidB#B zbM%}uF!6M3W~=1CfR{y<9K((X+Iz!Z)AG2Xw_1!dp1jYIs)pMSKQ1OcdV{hQQHUJ1 zCB^;}L7{yp5z~pz{=v%Fh|>^-pjROW;%xh1`H|Ez`2Oxwulfp($e^kbz4kBmJ`_HZ z=B}t7(P<{jFo*8$Hz3JmYqkd*+isgAn{-kN5f=p$AghbXJH}3H4`ppxu0P)}QUEvm z?F~^r!cKH6T?V1m9+)k~@*v-CwjsrfK1Q*o9Ci`DUH%&%e23)TpBOZROwf7lWV2mr z%g$~JM*Bj56txoh*_2kgw4FNtt%S3rBidPVDJ2J>LvCD};h25Qh?S)YBl(1Ae3b56 z?j(>PwC#{7<@S<^2HD>s2rsKNrvq`6+JA4QmikzzdQZU7NK6lD7Sn0>yq{~z>o83kd3fsnNx%BxY ztlv(RP}b2*a*~ioh=Rl!VVE+NBs!`hC}ahejUyW(_R!AQOs3N$*ry%LQ{mnYfp%M{ zJH{;&WHamaR}#!cGYizU4|YIDVAt!z*K5!%CUdIpMW_u7aLYn!5`n|TBJM~XCD&^= zV_by-o%(iHdmh%z*|pqW)3(&jeyW_;qLu~M9DP0RrPKk1I+GXGwJWXnXj=RbX;i0Q z6|N(0=Cw6ORD5S-X0)-nzCy5U?vsu>S1~U`1Bu-7uJaklCm7QSX-F|G>VvDo&uvOZ z(JR+}*=0@S{ktR0IKpVPGD)(pJb!RLU8Jt9mYuKa z!fMD5GmJJ#&7$9zR!mc4ubW{X`&y-p4|JGTNh`{nW(HQtosZ=->0`;I1!0Y<+)U3y z^ku`Vca4kz@~vlKP@`&gA99i0ka;iK*>lqR+Iv}mof>7pG#)Ka%zCVAT&w(xZjt5spCSbbGy<|vi7dchZx}d`suV%bK~mJY6Da{d*}Kr^^V1Zc)ynM zH*(V19S2V5p2#%#NTX-J%V>-}uC6tQ{Mgo^0T}}DIy@aT=^F4B-NaxlzM-r@x{|rS?vxX&JfcqRMS=CWo8jqA> z*?-fh%%I3ngaWK&iJd>(NpmxbRf@wb?LvQ|h#Ng0&deFo@Mc`pm>V2gVkoYDoCGU^ zT&TUDE+6W${8g^kX?w^Ae74X1K+q;J$9j893Z?+01=U^!CACJZSx&D7>f%h6z>pNR zXwk;&T_uC}FwdJR@q;e=sD-jEx~>#kt=3CSn;3`{1C(UM*PBdWUMYS`TZv;COx*+@ zQ3gZP-dB6&5g0x8Gk_B@Mz6A)I&PPkNKf;)lDW{?z>F_SZ$VBY#-t2-sjaMYbn0yN zBs+w4yvZ^0*&m@TrG*RR%CO=rzd>oOW@^JRiehPLpL4nvBZ!=_KyJCvL(Bp+rmDRm zOi^Q{Mrw2V0bARL7iRHm-0I$1ANqH5w($Ielqel;hN0k-`v7?)hPz?4v{K+xuVl1c^bzVafgJJtF-+Vx0uTYm5Px`U|~9b2IOqfP3ut zE(g6g1=(w=i?O~!&+>g*w*_6lq$RIJH6M!$Usx_T!{4~nTkXNz=~T^JYf2+jS;SRo z^q%R*cy7IMw!(fsC>598UqDHf*KL=CKbU01kZp;1}VrZwC;K1#r9=n!tnM(0E=9`!75^aT9OD2#5U(OOVGa^MMJv z_*ul#elrV80<#r-x~k!3vK*h2pAqQ7R2xvV{zc-i2>}tXmJ4cpC(_M2Z`S&{@=hj zG!_s2Bh6PLhWFuc1l|kr8!NL}i^rgNLgTRno-lYK=I`Zz{@y@<0P+$E7=UMD0ukhC zYXjmjCSr*^&qO3{-Yp0KH)h_P5Cl;8&2@nQ4z)Q92oS-|#vlj~c(iEz|9g=AoX>i> X`@yue;R#NfC=n13Q&BNGdIa_-*^pM@ diff --git a/rslt/bgt_cmp/Copenhagen.pdf b/rslt/bgt_cmp/Copenhagen.pdf index f7d9faa64165a811d01ef55dc95be59256ab551a..598ac16fe96b36f29ade0aee600b898e61ba2fca 100644 GIT binary patch delta 2393 zcmZuvdpwls9$%L^G=;F%rN(U|$>(_AnfING+mfM>ahFThH71c_P&5XyY7?_Fu7zOi~zvMD3`K$U8R&<``;gi44OB6dj(x$#DwsYa-uN%{=oG5je ziEqABV3s>UR+W8lYPc7dl_{g;t!)XN8K}3kO|A^u@<{nj zOMa_kTXtjbg`Ko*E<(3*oV^Wa{{Yk5u$vlG7c{DFby04i)brc5oLNSuD<;w}S>%0n zB>;WBmq6Sjxttv8Oe|VZw zOJpDqR~@Y$O=(`xwOxTm3H>b?Q))BC|d^EaQ?A-<0&CEN0o((U0zv! zJHxoUoZXQA$Fad&o0ucU64uq>O11~=TR$-j$$3I8b4{Z>BRNUIG3opZp89FSX7pPg=VIuL<=ujGKaT%R)SC~N5~}>i^4Fv2fB1Zg zu9-AI*X%hmC%jRADlqLOICejn%HbFF=Dq8&?tJd$_p^IMVjv+!HrBj%K7pETl1X)~ zozsp@svL09k1y!z(A(Sd+UK{OyQ1cJM*EzytrzBWtI1FAudIj+ z7=+4b=4~@qRilE3AzE>m@2j({8vPU&-150DhKdRes4@Fep)SoMellvk zu2skCyIE*W<<$;lvvZpg)cIHEYD7BCk9~w%{NFM=V*2SnOdFoqkXtIaqEI68W^Cw* zJRIXugYcw&$@lWzCr__+cC88os}Oy<;1D>IcbqMkk&lVmlbCS{J795~l1X{}lgPt5 zz0(zcveR{#=FSvZTz;0X8oxfp`8aqjO2MJ^AnylW&#Ojf^;xwQohovdvYd**RA~L{ zse_Z|W_IZXa?Zl?XV$9=NzHkT{S%+-@Hdmf+eRB>Vxh&EM2cHM_tBQ4b-A4e#*Gd- zALG1yb18I5tt4)hZ{EcFOLUFW)=*fr7n{N}EJ8~?e$LuEwGA`gFS`)Rui?*7-Lkr$ zU^<&PN85U}Nst(2Cw1F>2`=~bh>jlJtSP~M$g9r?Z2ap&M$-@Cyb{T9U z2}A4&Sf7DID1vwr5`##Hmp}kOU|0eKiMaqk5&?l>K?43OkMU0ERKK_hQR-?nDM1$2*M+VAp#yD3}Yej!~py^voM|@-XH)I!T%)wOJJBp z+{piR&;MIEfCm9A0)vnb;t4QOoJRm4glPg6K$s!Gg#T(V{uCwFR-{75P)&wi9vt>AZBs6ziQB;0}n?Wibk86 Su|E-Un;-zKr)T42i~cvkl8u}I delta 2451 zcmZuvc|2760*?yKSRbx?(`1A!c{(#^&YYPcqQupEmdVT0VrxM~8M|;s#N5h$Gh~lD z$}T)R9A3{T!{+@qV>+r48*xLp=4aT|_$}b6I)ulzg zY2!+_MSD%-HiJPYy;UB^d*eg#-5Fieml}eW+h1nl3lC`nv;U4lxO@+PzYYZ%gQDeedpm&VAsQKJzq5Nel&D6_Ofv;0TRCn?-plN2*p=5`;wf@h z+`LP31iv($pY;|0Swf+>iYfkR1#9wn7GCyE^Vj&xkx?m9uS{HrvEXf6^O~4Won-45 z>#{lBG<;mRRI=JSM&^`qfJxa*Vtb06a-+*V@>hA54ziP}igCyzr`u7Fr3|wFK8}@? ztzl`%`~!bU?vO-Kl9b`cD_E#9EKlL=6Ai~S)=Hdi#H48J_T1-L7$e@fh3F6yu%I^kj22;r^vPw;EONwQ62#H*S!> zgX0-+cWfp|Y{B$nU&kkWDx$h=Rhrm3WnB4;i*=HQ`QMt@Y?nDHOgep5Yoc0WN2&E4 zi^ImV?(9jI3zC=(!)!SVAMKmBPCSa_78SaxNL&QD9Aqg>JKH0T7yUDYP73ZU~<>Ae~Bc`Y8b>}jzY`}?g_@tFt5PsRS%-?ePxwAkV{o?YCC z3E8Gw&r^5pd&aqM-8KVJM?raPz2cH8pUgYS;$&n$T*T7I2|Czd?ly#*5;hz{q@0AuBV@SN9b<7x%R2S zI+eN_zC(&8|MA-3Im5-!>UjsBeM8de6HARBEX`!&HOKH)S;v+bPefLYV&$R+XX%q8 zLj}2~Sag{A(32_ssDVY^&5oh-s(R&=+9T|bT)@crib~J!qGiQsA@)By(Jamn(RqY- zvc~Y?TP&K0xUVSAnitlOqf;%;kI5u_v)NqK}4{ zUb+9_^DB!k(aaWO&S){eWU)ADQKfB8)2KEy-`=XCKf1Oq-!9BOt(vdGF)O^07pQGF z9MIEjUts1DKiR(nDah$}&hxD4?K3_;#)vlSbscI6uA%-_`Ol$L4@QorS9H1pL3OVS zC*ZRFjPKBc?1@8d12T{k)3)GcLs^5$H#G^>^4CW{O8K1;h#lCIA!ze^1K0N)`y>|2 z1tj@51RFAv`m0IoVlR6+lm!Arkyb$9fYg7JX1HOinx||2T%e&ii^cw8Bk%! z*GT!%1rcr%8RIDnS^uiP8p+4*e>-$JLw?)j}iwr6p<;HLLVXG2sIyVDzV)HBkp8uj>r zC;+FAS++8Hk%*y^p%cPHzW9c2uG#2a4A0w#buxUhGUZ&?Yny5mT(thh_Ic}q|LdaC zqESJw804upzxS-$z4S68$s_*ZE43erueY{1^1Rch`n=aTVl&vHGbTy6)ra-fP?l72 zAgQjqtS;t_V@vA6K~~9BAMK~`)uqo1X1K;KHq+DYgm_ys3(HKNTSr7ma zg%M#G6t*P-02(6#zeYj+zX3o5K{84VQG|EdvZa!Qc_2taX+a9wm_+#f_Q>}9BoY-R zCc#vcF+_xg8w1cEXdxI7UJ!sNq+c8VrZ5=*5aHVd$XjngxE&cnA4rBNurLn>U{q+B z2%^HkWZ*v@$p4<;mK05-0Ay5S3PeV&MfroZs6-O#ny6%SQKxE;@1ONt6T9EWx4)@7%&67G3ctue0?al52uo!0btg=_cj!JO+^r2q)4 zG;jyPW)>LjTkh+7YN&qHR=pZL_6*t0pI>vY|Mcp|_mW@ySse-5CP{g$wzK%$zAHl? zr{*&v$Gjf`D64Qj*qwiNqt&wsBUj>iZO$_rJ-Tf|XKHQ1YpVfA)})kzmT>pmg62@y z(O_35BRKc9?`BzfrbK`3x)igYA8Wjk_q-LLex^fY%1m5wKeKF)vGQpQ#ZYis{hD1< z_F;)*PS?Z|L9?R@`T@Kn>0iy}hu^~cqZm6o&a2$*Di zy~Qic?2cesM8;G9T*Zq78Hzb1p~ksBHqn^8$1S&>dTyJq9P?!Ah4Pa{t~#`J-8IvH z{1G+Qb1)~18GrHcY7J_4ND;cnW zC`gnD-mqPyV0E5z^7hzhwUB9-ie~qazh24$G4A5qMW(|qzmJ+(|7PG3LrF0&@=vxB z+^D{VWM$2T5y6{ZUPs()g*E7Pd2D)X*eog?#cR?LEv4U3>{Gsuj+3VYe?>YnJ zZ2n+Hmdp1Ef4CvzP`egLEZM421saS6n%=3lOgAtU=ZQaQwhXBc)m2QBz65XC+FPn2 z8dU9QAe@Joh&Gj1_fuJ~swK}fh8~Cel+v_8y&E&VUH7X@dZ-D*$LEHV7^NNdo%6k& z^itpBa*}dTE@8dr@npo-jES)X=~ek7Ow@VhO259;cWn`dUhHrXqrrvbI0KcmNNlh{ z+6nrRj*yuAc;oZk*NXH?)s1fI6+lGkFml#?S5{`G>RVREFuxT?ZF8(>kGA2Q(1%SA zyDk7igYzDD3y;Io0pzohmyV-IHH@ zE~{zD!`00mHrL49^fa-jL189zlA$XmtBV-cMQHan_!zu#8lO1JXPT8U9l2ZANsE-U zYHd=WXHwv;v4OP{%!tgcNP>Cp^(&|JwZ;cl*F9R@AD_k#!h#Z*dJC8N(klxssz$Q1 z<(pDm-78$2PsU&jocyc?7;#D2qnZk518#yo6B!Qnse>H-Qi1u}$9l$wwassyvq>7( z!6Fvu{VpiKVzQ!cTobbG?xZ_t%Osr%Q$vkh#Jd(vbd`_b%cHz=s~ z#(q;f@A?8PjVHEFdWX}*s0A(Dm{LY@Z%zTx`q0ri10~908SMgQZZueEqN1s*R+P7dcdZA!vJ$mT)@kNyb+Kr zR;)!&ucJ!J6Fhf6PyG=2#GYww88(@rd4M@3E}-X!^tLJFzP#OL;G}wJcUCkzr7>+2 z-N8j_BDK*Y)l*vf`-Q|O=_Xc!h@6j#_~|W=&iU4DBg}uuBOAEy*e7w_)N;RVO2e~q zpBn=@V7(EdxA1a@3g)$4{{EuSsLg|(`}+C3`dO?7B0F|6&uMkCHa7cWM>gh_U7W5) zqu+g0JL-&^0hrY7c z#dM8yvk|>n3SO#LSCq+aSj=Zieuo6DbVfUa&e(nOwA(pSH1VeErJt9h{)PgBIOqo5 zGY>7-#pUlUdNo`W+~EOL9hCptDYE+gGx?OrHZR}t%g4M&MfQE23T&qd6b{uFMwz8N z@DfsC2WhjZg{Na(1ryp7>$rhq!{V$P*;~kV)}y#BP29kL)Qo6aUqGi(zQurn*~5&X z+%AF0A*K8wBYDt7j<%(#Z*(G7)m)=}IMaibP4zxcfns-etdMuJ<;)a3~c$74gn z9U3s3X3R$7kX>1*1~W`Um28&~WMuoR^vPVC263CiqkI)!d9O|v#dLVMc^RE>{%ThS zE(UbY_8Sr&cp(;hWoM=Lo^>qw2V6GSP;7D}Fm%vS88kvroD*gQ`EZ!4`;pc;n!cURyIdV1z*`G4(! z(M%MN>f#wtH<8Tg8eV5D;Ig5wG5h+}$%v_uve2!}ux+YS21?&AC&CDN9bNbM5)=*YnaHS^IcK^dcm7QVl!e$m(2IiD2Pt{G?jwAL$0L1FB?hu_=z zKf#r2s&{kIZITC%`oC){`82tyU}(kg#FVP7w{_(;hE8)L?yC8C)1l9Q+G`C8aXMZ; z6s>5Jn!o%pFRI{tsz{HYLKwc}NX@uHi%duTz(l{hS=_hE?$J@kk2$8_IijTb(}5o) zS>A2ybbRkePE|T@q$r#Wc>DC@-H>Vb3N76);M>Ds$&9aiwNTZjH!}QEqWVEQEg2mT zjy48;3(~S<9FrH79Qz{7_OBTT8ljKsIXHBUMd$7otT67|EZccI?X}qfcY;OPj4{zJ zIG43Y46HEz=x%mpJi|PHR8Oyi!+4OsfJi(esg9)4kGr#3G^f+N8ROU=NF0E|q9E)V zK|U&pLxF!lfP_cs>LM52yl)cszd2ow73TUOHTIXywtu_h-Jdq+V4~KqD5hvKe|OOF z#TthVG&vv9QW}xq2@1Ih>=oa7K}SejsJ5yjYS>unZ1|wmq=5)ujSQ>p`d{0icXh_P zgCg-c7$@VPaHxuTq2qmrq?DUOR&~8gVmX-ftpxc|Fh@Tof)IzM`X3i>DYPsU(?mLL zH;cQOEVLK>qW@zesvjtnXgam={gs;SN4vPivw9fn*rxPo+jjEHE=aF9^?bd;M(W1L zRRvDNo4@X?Don<77Vr9M7FC_H&N_Ft(V7!Id5NMJ=iOYaYf|VVV36QG@YQj%iM`7P zt{>mU$62JU+YNiGFh6z=98S_pE_Tx5bgnHB2LUSSH#kNI_FU%Qxr#f5nclIb$)*cB zKnSZ@c<5{y@p=FAt!2`dL_^vgoue7CX$QB>kruvTZR$Xh`7vE))tRFaA?Mhz1 zFSTr^D9io!`>JDi{Y(7qpIYlbPW`JB-NFdB9mBS_Djej$8jalbR;Bw+fbRVKN*9DS z!R5(ru1$pcc$tpDp?#aGl2MfX215qs<2FgoEzd=T^W%juoowtr3G<_T@45V;JJ%ga z7CnL_r9~fohK^7@Vq0p$`q-=>s(oe@+Nro5ibIt^6eTp|ugHaPThus?O7#dxRv8W1 zY$=0URMa>I%2yGP1sa{JYJq?rpw%EhG!6<Su5RUvD zMB&Nsv9~~JGM!j7`YD6mizXV&nMP-$2-v^ZEEEC9n;|vbQ3T%o zffKpAd7}vU#W~Z-nT8?|{$9ax#KoB_iw4_CDALamwig3M_*n;1(-TEl)b~OW7G-@< z#6>}08jFD<;y8&E5d;E&vZ#uya0DzKj{}HM6V(@ihYo0If<%CKnRx?6ME#LfV@0J;!i5@?EkxgB~nO#SNJa*3C}xbyz$GA^dGS#GWjNX7%CWqzdpNJynDqd)=3OScJ2AuQs@g1id|;I;ee&Ia5&h J*~oks@_&$G7o`9I delta 3937 zcmZuxc|4T+7rw47t}Rs5l_4~Qd1s$NDqAR9a#gsN3|Sf(!&qW&6xmX}_CZ&-v|I@z z#aNO^M4>3ln8;Pe775vY@04yo-0xrC^L)-Z&v~Bnec!S(d^f`jpYhRqOg2|c*-uD6 z)PK<1r}#wdvfPHxg=QtM4U2^hFDno)Cz{5~&CNJ8XN*ORi+gIyA5c*7X{>Xb9^ks+ zV?36CjJd`9A<5Y}J;t1P1lwxsm9adjVhti^ynE^XeajD*e}-A?Sq z9ei{1kLB~nQVLDwQz-MN>D_z6M(t2PNh~adw9W^3p!B^+%_9}I>Mb*!Q`JLBQ(V7M ziY3YSWBs-FGD7aibnmFz_QO}r*&pQou}ALrNzs!3oPfN|Nz;<8zWxBc{g}F}@aYCX zE|1o3HMQJVYWCZ?5l72MG)oP^fv8PY!sKkb)=kXADer;P=z$K;b9C(*vvj|0*`fs6 zWDmaPjzr@oC&&EYG}j4_((LI*le^i2RoS~5b>nk}szk_~>$4Ya`WJLg`5&Hm%=&%}b! z-P~Je%s!M%KCts_eJc{pH^DTS0~9iLYETY;Tk#ha1N$BCKJfsJf= zkBmp{u`l;#1}_YP;zzGNeHfCNWt!2PaQB9g+V!XAL$by1PewXyv)f=cuW~55zgynf zLR=2ffMUc2J(_)J;}qQ7LC8Ug^*x&n^@dp}mt#B*TR~lldZLarzKMnCAteF)A^rRb z?>NRuGV~sStX?0@Ek-TY&_cLnXY`7MObYX4c3VztRQOwI3Y*&Q+AE;s&6}!cZ_e3k z7oW7mjj0iygEB;B#>eTNhb51tYon#YrP?W+4k35voAZY+p#}O{X1&QS$A%-FC~zAp zzDAskRT3Jid?p|1=W%*B-bHUmvtu%&pN~O5FCFG2`A*&NW3w+Lu}#3!RNZ6;v@xOi zrzZgUcvJZ}UIq4|ns)z45dkvtt6f(2eTNi1(e&40b|GeZF>fTg?^d&xxc^Lk8N##X zCeLVD`yOm`)AVingHPi|pt*664F4%E<#T`PK;$2+ zNTlSHkv@u<>$^pY(KQ2U3r9q=5PDCyCe-fZYHZcUhCb4?>SDKh<4B@|QL`7WE(%Z+ z?G=qP_yd>p7GwKDBd!m1ZN7QSvgKt*!NL4nh5LV*8tg4$r8_VG;qqy!jy34jSNG}@ zcfp`KYZ>jlKnSNJ7csdvKheM77_wV&B3&n?N3xn7z2}A>=j(FzKPY*E+s_JQ2q@!( zb$lg85-hgyk0mw4W(_ZKk@}05tlE~Ecb<#3iVV6UpgYhW64mYcX*%TRypH3i>vvpn z*TG$p-KsCr6)ruJVCDCsRyFSn(5Bbupx&iXK9oLhp zM`SLUB(`lgC?WRzo^Z#kKG&Cu;nMU3*GU45n*O_S7ax(IP^PeN= zYvJ>l{$f?e$9m5^ zs>nj;`CqkvYJIIBk`({&1ZDI4sLQrh?hy)(r4Ek5i<*LnqjMau8p#G*QMU21nwa`A z({$OaHcQzG7jd~yOw1CKI(vjzT8?=qBB-bw6DgC}duakr`0LJ=9=ge2C8f{X-hL*E zjtzV3xF4E`*30D>zOz1cwMbWbyP#*ubR;WN=g1ZP{WZM-vI`^H9RI;F=@@UxbJ>R# z?&V#Rxo*OCX`Rw^?caZkYcL7`3o^S?^A7wb#8>pz#$&?F<(KI{j{*yv61-u+-h&{D zk@sh*(UbbR5Jr*WhpPhgfilZ7i{CD@$otCAUfl6&H2sBYPj1swi%yqaqeU(vMl=`4 zGzzN#Msv#2J_6qIjXFE*B4J}P+9Y#h*dEK?*m@?`SoCes(S2j()4Kd4&&7VUz{8Tv zJuD6Y1qaHU>@PCNCFzN0u-Y{bvt3ifAN%50UB5oVft?dfb-rjr}h5 zWj;;h`{;FInq8A@g_YWqT7P57x#ytt*|<~+qdD-9_8Js*_iq;G|j~z@qCEetWssk}&X!}d_`kRA?-+^v2l8n}CY@aW5_bVI^4nG*TjnkX zk4tdE2LYi_9!(y4F``@Z~e@Zjf+3qO;ru4_R62H zvkx3PvAfN*^N#Z(4$na7-Ci16YWNrMe^yr{BR$a1(>5e0u z)XK-x+aHYnlsY>vu0_0kylG3e&)w9rOQ)U+dow6vn%GQ z^e0s}Gna%(IZ}C|B(i$|l|pho$-DWvd5a=ZFaXB8pwI-oDAV8K!ZFS?Q# zoQ1v6Bwh=M8II>f?r8Wz%aJvf{IJw%ZcafeiZM=)@flDQsy5}s&itO3xve_BwB~m) z|1c#Qb z>7dDvyJ=Lb{jCL?j!+sj1=Eb%NlJ~C+)%L8-)1?DUA>L9I3?`;3UdoAl9OTjNnDo( z&nZ`fPRbe}c}E#&xI>+3t%R^wvOu z1{&&Mokl*Gr2%JV?DFFS{dd8cfR-*FxIqgJg4%S@M;i|6Ad10_2n}#E&<&0v;7oaB z6Q3XffFZ2G4=6n*96c|{XAPJD2d_+!vai(jszbc61=Uev;iO@(I_+u1Geji@}WRW zJp>(t2Ovt`fstXNUoX&e0H7XgG2YN?FdR?PS`7Cec?3x0syrybL6QJ)K~IGb0F`#D z(Xs0|(E!hcRdq-dfoIqn42^<(SdC!-BCkJm{?!Gbh)}c!L!*$8bI_sxUm_aKGm&?G z6r-`YwIl!rz^^qAz@SmYw4v$~w0*;8JuTz2t@aq)gk;HY*;IZo);tAMwE)b9y=;5v&nt=Yk68aCt1T1>( z8v+RUb?*ZpqQ1T%sN;`XB8IqbXd;pKS*;>Q0?^`DVUYhw0E1i&@-Q4~)t;a>NT_;U z9uiGhs}708V%HHP@%Xi?L!zK*WLHfCkznvRH0ZVW*Ka=y7~+th9~B1C@_rV!UZEry c7WM6~?Mk5rQ%D}7NDL7Lh^nd@TbYRd51Qjwxc~qF diff --git a/rslt/dist_cor/dist-cor-stg.pdf b/rslt/dist_cor/dist-cor-stg.pdf index 8df46a5838cdc07418aa6b60ab54bb3007fc7a90..4d6ab406a34346663c30884c0589a1d9cea78970 100644 GIT binary patch delta 4407 zcmZuyc_5T~7j6hQTtq4rMj6I7`rDGL0#t(46$L0LYbrG?_LuLO}hVGkh0#eqYnGgXb8=bu|SbDGC9-FCbj zurXB_3S1}D>S4r;amSFEv4b3?Tk9PXH~()m4lBKUscu^H#`&AhpL7ac6I|AZ-t%o0 zpIPqDt#@#mE_E3xasiV>Hb#0^Djc)$f0!%dvYW5JffVEfJT&=mTev^0nB6wABiTEc z|ERhL!-Nn{f_y|Fv?q1uBZ&Xm*qu|wnRi^h_djoS-{k48v{A!1{&k#H%)fmj)Pv)|)x z_r(O(GmdCC(bTh4qR%#`N4{P-TSM{T&w~Bt&2Peb z%C$BuSky{{)JauR5^JhOhH)sJcDf9|!EKkr@_!IV3x`6^&>G!~ghAucoXVzcC`1qI9rfzBIQ zf;$qY+uzuX#9Td(xtvZf9v~xXHF@|?3PJ|vZ+Sswb zfgRBenNZQG3O8$C>Q!ddD(kV#g6~ehasPnE)rcX@J92>rY>#u+R7^#`48U@>CUmkYp#Ekt2{L$ zcJ1=*;Tp%zc*`%{BdOzy^(=e&QG2_uZ=*VFK3OUB^Dm9I%fp>U$H7a2ia=mDM=#r& z{Xu6qCG5EB{a&^CbEOL;vbTS&iqh3Zr(BorA#|HTT6AUERD?=MdF+*~hcXn-5lhQD zdh=&5u#*N8tY@1d6ZFN1fo>Bwg5E0q785TpmpFVV&7e=n-XUbs!9IE5akKv{?R{7cOP+Y*xoZ+Y!tm- z^-On~tYGkRa^Q(pm|D`MCm%@I;0)m5m2)pLNSrZGTC#E2i{XaM^fc|}>48E1OMh^4{%vpmE;|Sqj{7MD|4r~ZL!UIO1=jx zQWb4{^zhTUWNrNn2`wquK~M9ozO||!UmWUeVCwkT_&OYjI&DmeDDrxub>9F7XjCO5 zM#XFmrV2OE2!_r1qF`a1CPTVl+|{PSJ*>ne`sGN^pMTft<$WzA*7v~6QuMa?QkaA{+{Aaaws|F*;c;WO z!N#^tvW3@r&QuJZ*1g7+w&IL0duLzKdKF8jh_pnpM)yR+<|0vx`=KU}BD9+D4zxv{ z?01Z+e6CMLh^F#Qmw#ww0aUcI!GB%0-4f+0|6szl@}DyiNX?yjai6%6?E?V!XIm;DIaOo-vQn9RoSsBZ*^f^QUo)DHn&17Lru`VumW-#_F zJt)x4rS)U-HZ#AofT-5j4B*RqdoMY}`>8j}T0H>;#a}LzE^-!X<&V!(8pORH4)_Nd zY^3Adrmy@lAJKh&z-wK8M}LAxSDs)lr^aVe!{RH!`vg_2zd9*^QLo3Db5hS3@a21Y z<%w_HIL-h45bd6C{hg)Dq|5yAb~iJ93gdK_0svs9pVh5qr}t?tRu(ABvwXiDa}yFO z@|u|uRG*zo;b=CUAuYvax14DT{nQbgMeLv_4nE$oE$!UKYch<y*A$g!c;F89|oQ3=*f$`o!o`d7$^$L3ouwRJ_j!%$Om8u zSg4MU@CuF@hd*5JS&b4cIxmbkT>hndCYd>wvS@FjGN!Om!Gx3c&Y?=gehYQGpLi)1 zk2?zpxd|MQRK2DpBq{W~t|x5LNIKx^JLx$+F}@}l`l~x1)xqW$Mmq1rqO;W;jV@jV z>uA?{+IlugyV*ZzY;ldR1`=jUgkKA0?~7pLqTpnI7s<{7i-L_vVMlG#sJKL-1MsS` z*YVIXOo3Fphiu!;CT*L(qT(;QYUF9Td*LU#iBI}K-QuL+7I|5+?Blw8$H`q2d+Q1k z)q0B!CrrZ{|B|3Q`tp*J6FzsHq!7jIDAqA4@DtFB@p(Jpu*<~GbxYBCzL3Wc2&&rp zLv?CP``;c<&`m6M+zmEMN##ubo+1fg$b_E^GB{#2&_ZO`a?<)q2$m`ece5M`Ikfc<19-9SF0eDA-5STf7%3Cgqwq4^ zCt%Y^;*$xOX=qcZ;lSOo7+7vsNp>@z_vyxDn5qutMAcWB1WJbUF~h{o2b0S8iqn1m zY^!rh^DptYE2Hdtob<65-syey#5AhAOa3VTt#Dz(2BrSw3pyOmdRK@h&NX*`ku_Um zwp>fk@}OK@B9wGkZ(Pr8);httvr=3%FIq^gmw`GcWoEU8)SHdb$+)9{`5^hrg zXDeHsC^ezK?vNMPJpCTjQP2h>p(P-GCmi%sC;}0RDmk`0TOi;T7#yT1mV<3dDmi;# zw;-Sy9G>%4$s7Wv!&N{g91RAjz`8;qSBqIUJNJ>_5Ga=#i4nTFwzH#<6R_fBOf;= z6o*}jX^zfRC=U1ija!FbiB8d}+&lqF_*TO3^@ifU=@3Sqh2mEDeWAD&T0ba$h437e z?hVDGb7Dzi5C{gmPgW2_<4{;E3T!6(L$ILfZX|%mpt+)W57Y}P{4GG^QD|=e)j3}O zRXH?|eRYoT3mloN=qDWag}@yT1#))p;v4IwL}yaSThL`gIZ%D z5sTrS8~6ElUr8zoT@2;m%**`+V>FKF|BS-<;eK)Xk&!Yf%t&W#M>6x5c7wdlS#L z>bu648r_eux>lwfZy+DvUm53VVpm#O+**;M+FJ1`8tY;&+L*|(3K$u4iSs1asHi2h z)*qZY4jI-1#g8`*sYhKNOqGvH!*y~@4|uAKIBF3 zR}dsC+xI3)h*>kN)I_}*$;_Ma$;9-P4@DQHb}-X9-!BMS1bcXA8~Wksb160-O6SX00u|WefQ!+#-cR?bC<3``lNPpmWa;CBYrX_| zUKw#{*t)?ffXY{LqD(j(wFUVjwb2<}A|K4e)&5*2W}6~Cctt)r9D6&#dDu1k4MRx% zW{z5$jiYS@QStt#Hm$JRHsOLs1*VlmQ~2Bc4iCz1Ly~No1^npL*VWZx9+`Pdq@EBW z{+3M(ahDnXJJwNK(`qT#owJrjAr_^qrfEDd{;TT2trpiqNx?<;Vy^sZMsnh%>Fn-1 z@B2jQb5xnR{Uw6pj9!nuiF=oJ*(NiZYlkV5HZ{lG#ZuP|&k8=}cBAx4P%_jm{Qw&* zS(cXE{>8QYji1oRQq*b<>>Xt`&9uU!GWbkq@LTFt>71JO9uu#{Chfo1Tpf1$Tkc6k#mX%Y>@n0a#AkB)~*Eh zlA?Z zEUJ4I&K)ZaBaA+H**xN+@jkfSY$YhkwA=ll`}scWlF=Ij1{toqKWwL;8lvgyl#Cx5 z*s((K*2k_M>+g+q8U0P~79a5Nwer5#oS;_T`>eaO`C&lr+aJJt51F3 zpA;oe6Pe#>M*Il3>W)^Wx{Oz; zj#f3a!_MqrkrL%}jhu+MVO&yYrJ6+qXrO@bnP;85Z|Fx^mT42V*txZW`8hJK{LHBG z6St3T=-dnMMF7uw%}K#P#hrqjaY(L>dk8v+`g_maEQoyDApXm zOKW*cMBxe<3)4qOQ24GWDcOe~UABRFxp-HLYy)M6W7uhp?|reBe=c#_Q0lf%;u-(T zlvAMK`8YP_H01G2k8wiP<=YeS)T=8Gs|5SnH;I1J7S1ROlRxxW zs*-^=Gzofa<`THR23_Z_j&|~ul7|MeNK;wXBZHcKSKQh3$FvCx7g z<#yxRN`ooa+lmHKU7Ur7DrM2z)<%R9&osvYat;lzmkrwYwoQGaqX#cvZ)01d&T5Uv z(rlAf9JD*{QER4de&+Crq!hLoY;@CW%_Dxv@Z}Xg2lDr`Y-8n>+2yfv+0b^WM7anR zo`g~25;3c~{>0PoWR;|J40A)f;#8jk7yFVMS>(gryDYi#79IIQjQnVe{P?B-)Y<1K zs&-tDj9LIauv>Gqp{&|e^kntHt!Xl_iEx0Y^*iIz-ca_nurw>QK=Myh<$SRug?e^L zfM}|}0@oj#*Y`)wbvu(61jw?NGuW{6eR`5&T{&(w*V%^a7m>a;usK?5k3?pm7@VK1 z=P!4-JDd60^`C+;4aah4+ecXgGO!{-weu? zGEeLKk2suC-|_6<>NQxmdfvTm$MC8F8K~=^qt#%3k%3F)wLpbm)7(pKqn{nu;h)nH z#HX(hY(CCt-1V$L{HK__zK%eTPrqe5UOP3ev&sB?!g|eHE-#{^jt(Q2so8bWdixRO zA}+8TuNt^noqI#8%5fEY@1t#>b!NvL=RtQL<%a#}t!Gy^ke{gU?;CiM?b1>p{>$pd z0$DMWA;K@8`m~kDuOr&c9P5@t{1q;U>ui4B*KAjOMP9i= zQ_<-bR)TIb@WbOMB2b1w?tgh~JM0fF{?|riuIK&N&+{x2UUoKJy;JF!)AmvXe}oI9 zRp%M+j*IXjtNy-%fXKDa3WVZsN7GHP0d6Z@j4vRSHY|tu^Y4APmO>0^WaxlB`x+ZIxkT-rh#e1WH{&3gJ$$?RRlom~*nUmsA zPki|HG4@DgpMj6wqNa`^sfYdcLiWd=<$d-@I~JF-K__CULu})d(f%TyRNt;)`w+oI zyIx%Yt!Uhr_LR>#2jOE{B&F_cc|&(BXHFFr6if|G)?zcR2KCg&FyGwfieIc~6A$yz zhju;b+|$@Nsnn&b2tD1?qsuOzs2J<&)1q%5p-4*D@}^70Hy?4z;xNRxLSxwm1vF(` z#bv_w{JNfyL9anWM;`C|IyrB2!`87R1^!h&Jx0urWD{N;XICg3(0PILU2Gle4c*{5 zbhPmBc+b?f@7FvgRCWhT+n`xq`~CbmFmBn(#WYfu(K3<65J)0p3=Po;%yRwgHh64w+*Wj}Tj_YG0l z^wvlF`VzKLU$}wla=7B1vvHU7&TZ2sQppw_s;?A#DN>2YRt-$qJ=`60!M)8__jRDc z&_2y#wL2RyW5-lGx58Z07cAAMyuKhf+jqw_x21SZ!Q@9#k5{BLc*k(PkkQ$A#sE z#Ppvr2SWmWp8PN*aOQ-@`?(*&kO;G=kFyI4Ln8i+{4pfb?8%M8LU8hI6zGj1X;Xd% z2Qj1{{Ahe2hBQli7(<#R3}SJ-F=QYsQCAX$0^u}0RS|#=5(xyjK`$6ZfUWfL5Qz#P zqGo;=FAVy}0eK7n(tkcSGuAweGBf>rjQk&YR7BC7JS3nX0|D?i{dJ3}R2XBR29Xz# z5`h_?<`jYi+Kiv`Fd_l*YA!|s=!ma#Fn~a#&le033E+$n0D%7Q06-*?=8FM{WXgOJ zfCNzIy9baEgXa^Is0$q+BfI!L51J7{CJ}$C`ArO&y1-_DLICEw08sv;mO@$JAV8r~ z7Pvs6gUI|d3VtJ|0@MW)Q$hLyXQYFKjx47T$ulM(!=Y5~&ea`Rv&b5;Tr(ky*VPLIPadTdx@rscQ zo6@iEGNp}HZiETc;+c2-k%s%OnCt@(hLfeuW9<{6a#~#FzeL2YKCDho9wp@+E%Fc^ zijjOG=dJ}-fleNYJ$qqWsT1_>SIJvD!ogG_ z>e>~QzoG=0OWcE5KynYcXf+OPnh0kp~+nb>Ce{9y4N_Ogt+*X8aa~_aisj+O}caL zSI`f4zUtXdo<8{DHozs-*m+gW4C@lhwfW?P~skn(H=^RUC;#+Z#6v z5opT+t2%Fw5CVa2V;^qd6_(|$)MxBfwZ786ffa%q7|TCZN0!0#@1g@T7ihL_0GRpi)i|ASpD#$YUb%=Z*=*YyHEN?^nIPvR?e#*mX2(Yte~Z( z)+IGY}RhFZpnS{%zm)_8l%F&_uza7cEPGigs-{q zRM0bn*PT5H)yhcrQ%X7je9|--?`qv6gw~ddIranI@ zQ)B#Tu&@4RDZ(s2e-oz1l_FzchPkYo+lq_VMCY1=t!izSR*#jvEC-b!?lIfTL%g;9} z5lznytGlN8wKw)OW5CW61e%d{3`f?dly`0$*&Vl1z~YWZ+Zb0yzhzB=N8Dy1nT)Bn zA!_RaM&VsZ`Q1yTRB;nSXZB`ob5@MQ!_ml{g{YKJYE7L1{l+IX^;_F4=3tTyb>WsX zwPM#UeiUxbKgS>+ajd>-9H|3)?EgTlIN32tZxH!t;4K+e8FE5=mGJ2NvAMP2>%T96 zmmM6OO?{@H+H(QZDpS!-B}M7Sr9XC*R*YAKOoaxu1u%3Lv`g>h4TFu+J?3CxVrb&s z8}oC=Q)CoUL}ZDG<4x{4w?lkfU9+^SuW3Cnnjm^(&$TPYdrKUzQGO@xfIpbZ@cH<8 zeR^Nrwxi$PL3(eXE0xfEMx|fKyPzNx|dG1hvtX+Y`2+u`49C^`ydyO5TUXy zcT@`Y#!f1x4AvxIpOI$)Mt|a%dixG4e${IszQc4#=KYWIkMB;ufH2BCY%MliFQj zrI?i43VCNVd?Ux4ow9oGCyEPd%!05Wv+{654YKEWbeLT^WkSE4G0=bYWNZx>D|$V_ zEzfvw+J3qOWqm3ikkfnm!67=k?BLj;tjaFq^_>N+EVbNT*&u^*qc;}x?0hYW?9*#r z2bZ5TmZhdY(l1{yUOzvM?)4npA`$KdRhZqfyiFO=&jI}CEguzJByPL_eXbnexbZ7n z0By?7je<*54m{a-uVTZfE~kGovjAoYA+OXYIUd-5;_5(6wXiLHCsrS!?0?yxo2x9= z6)_^=W<(8e-p6Jb*~W3vy*8S?r7^Jk+y^4fm}Obgx-}#8#_;vbjR-+%O}@CfTA0ZB~bEiQP2XWwEjERN#SM zpK??V%5-xDEpJvS?$?!7)zQ9sdfc(%MQ{uwJXb-?u~WL=%<9w6>My(h@Id}Y)TQJx zrjXj-dFNKW=;~?B4UK^uu|vFGFPA>+O~fUC?j2<@N2+6Acj62htc*tw#NAO0=$h`<7ii&% zKt-uqEUCgV&B%@g)c+yJ-(nCVc(N^N)VPE<`FRW>b{?PmI4SCTZNG>KaE|x<#bD=^ zTXl{ksm{xP@{Ia1_Pe6Mqth)7i`qczH+LN9UT4@P#*1*hf@E$zST3#ZH|eVKXnYO2j?fJ;Z1uBP(?DP_91;GC+a73 z=%f|YvG)NF#&4Wb?+Lt9;&QWR*e4*VGXG&~%w6p&zr5-qsd=;Bz^zlV!6f4kStJQ#J<8v@9J@-i_AbV}QF99UZ@ zTAPlu0X3VG^1lXu{5)hYYE^tmPtaOJ7dZ2DOJ^?Cs$y_giH{f%E!;3QEE3=3ADqGM z_|p869l*@$Pk?R83EksHW|7+guV{k?I>j=QopFb18NJgjYh3&K-nr-3J7&`5-c^%q zo_||GLrZeuG~dJiEGEsxhadU5dcgrK@(&D>A7OD64t|ZW0Ei{xkUMt37jc291dGQ$ z)ff@(2{_uKY_8*DR^Z!fU!06p-pa3&H=e&S>Rh?TNt(7fNbC`ffIkijd#>6iu{}at zSVFkDq4VN|p;YL(QK@M?QGq4|^L5e(HK^sO;f_(!#C%m3!&B#=26{wiYv+0?Pp8|B z&)n0h!Ss(M@IImZ-ErsfY!oHru*9pv1BL4};4V65?9>e5eW=Q}eQC(I*h0xRZ`szE zCLL-ww|H)+DrIU*cC14Osj3^=S)6(HnVc*|_I|@dmx-<8x($UHs$Inf<7O8duZr8= zn|oo)iJiVcmS+dPEZ$*U7_>?+*?(l*d8_ducWLekfyn!}iQ9Gb2O3nr?j1RpzB8lP zMH6a~l;ljTy(R%-Da79l+Oyv%T2Oa2cSdz~mtvcJH?$ug-t;Zc?ZEjVi;C@K8xIil zX<6FT>+#pD)c($od|DeHkSP^cIGC?QMWnGaMawpfnuusWp7N1eN!rz*|7c%>){*OD zQAHXTzh$nRfZ=E&C{OFbNYjm`Eb!@y%b-GBn?3&gsdxmtQ~pJ)f9U$bjH+=Zv#6#h zgWk(;la(H{m*lqy_%Rx@l(z4%b*NlGq}%4Y*%)MOFv%>_6=V8eX>D-55mFLzsML1% z{mc(ts8@dH9HuZG?Q&Lv39)d4C(1opCwI)xuXKlL;oTqX<=UTD|5T=}XKk{jAp=Ra z&>Pb;|74%;_Nqcmq##jPwTp!@ku;|Yv~mT*bj9s5#O&b2Yq8XF)cI|84v&nPeJ|z2 zv>5N99r8L*JhB8LD4?Jqc`k%jRN*)(JcB{fN+`%yu?%WeR^jYYN`OJ;C=_Qw*%Aij zpj4nh6aa;)pr97iZAeNL#mQFjgF%66D2|S*77UV8M?o2CAT+100vW33LknmXNCDjm z4 zj<6UVVbXYcB9i!>n-$=P#DCWT*Eo*EFY*T<@r$%UNWvmvFpcSlBmkTgvM3COg>F*h zR|9wq4hIkb=s6`AhJ(yBH9!Iu;NkfT>5GJa4*&uNATH1Gxt8DnzLKRm;=ka?Jaxa} zcpD;bKnygmxfMo+kXqY7;xbMgfp6JwbpQ^>H~9A)mW*A7!xJbxlYhfuuoTQvy%-!8 z%h!Ryfd3f4Gl{TN31)D}@r(9jz<(N&Nt9(I6wFWHKR61Ou*@wAiLh*93Wc(i7{K6{ILPxK zz>tBZlK>bBW?2pZfTeB$0AZOx9%`8j02Z_SqrqZ{#K0xC0$2ie=^^BCkbbe1#dPuX zrZM5#+DHu}Uw7WuzId-#`uVYtypj2r)dA078j|PJ_rJ6YlNHFMxx)bh9!r8NDeW;g Gfd3ba{TCSk delta 3858 zcmZuxc_36>8*eI+gx8YF(%6?Yb7!AnL}<|>qIk=` ztEnvIElFfe$o|PviEk#=SG?!1bAIPM&u=;B-hv=lTEM*q7)aPqI+Hs*-@k-}xYjhBa?MAMAzYM)eea+4K1xt}6bUF-!Z2@Er{i7A)pVYfJaQ=sDd z$Q*1TAldnB$U?`y4M*OYp*C1iYAd|BomGA=9+5(a=6hz`*o3nA4V`m=a|A1x#(+WIdO^#Ev%FQg=;O}&K=WP+1bd7 zBwjLncBjTjAY>9L*xW6$TamW5#Xv%F+(Y~8(STFpR?;yf@-?CLNmmfEiCe9k+eKtD z=~~AgV#*UX6Sv+qQpOot=3gFH#Tnji&AnyHC1A=GWKWP~u$C8W-!+WK$UfR7+f8&` zQyxDt)m}*suTt?)1cRPQR>KxT6Ws6W9N)OO<2HDAZQ((g=3Ws_Am)`(j(NXGO#zQY zw(zg}PYaJ;IH1?)9i#4{IEjmqbkt0wjKj5kSUST^jE_X*eps`ai_Jn5fM#J>b5`B@;TT#s@%T?^ZY8PK zNme|~55=nHCU3Zebv4j(GuEV=oEu23y`huTeNslkQUft?#d+5Caf{Wem$k6XnyOi% z$B6=5_geppeg1C~qh9~aK!I74d){HE zd)s&TB@LbUI>iKAz;}lo7v@|EN4B492&c3X?+l03r-%_^UT67vRQsS&i*oW0Ng$tzx!<^|l0DAhd0^tTcw9VxuDGomK(^#SeT zzu!pSYd?PGc~PEA6oP)BRoDlD(Q3VeAN7DXs)_EX#sO&ju-%z&>tANay;yJXAs8e3q+foCQf zypTm7A;WUPx!k+@7PXDGb=AIjgl%wSn&~LEjFWX$H8-w;i9q(;~9iuc@-} zx#CsFY#XIgQ9n(UnDg#4D;*nUwy2Fi<|G|u(2vx|(9V$4i-n}EX4@3w^Mp$6&NbC; zPT@?7fcFT6@&rq8J`3RjrAFjY)tcD(qceNRx;Yup%-8-nz_FywSSt?NOY z_$uTq4D{Pu#o2=-x(!E$9;zaZ7*|ca=}8HQc?`ygW?Xa0rR+|&@RlG+CGihrb_G9l z^p34OIFy`L^Ug=Az2Kd%c6OJ%pJA2BdkgRM`??b8!Ry|#s{$zU?bo;OuKMO9H8-r; zMekFP;J8E8=2@0^NrT3jC@=4qliJP_w_lwEQ%`T)IKLNGkWR7LB@o_CcNID0yl(2o zA^L1n+lpHzQ7_)t=Vad2Yntfl3E9e0*pwmy6*U$)*gzc5N50|rRdRHJw7SAuTc0|} zjBChy*GR-8oGoL0I5|pU4s(SAH@sU#H4hxUeEUY^NH}4{BcUr^!>L6m3gLjHDK^*n1#;+A!|NlLH|zM=4q zAb-<+sD9BBfIrk!B4Sk>K`Fl6eu}=vX6V?MgxV<*b3lvHS!Z{Fa)RHwD7CLWC+n$^ z4AYi{|4>#3{_?6vh&K3ys98*6w7Bshcfgn@?IUCK#ew%XON{>-a|m8^&T+-%?Ba9X ziJ;_NCO6_5yQUBDN?#3z(RuMR9g>1H*YNhC;v`*P>7t?Z;0(#AGD_U5zV@40PaI@o zUANr*nK`&~v{)bm&uT3RHF0Q^FFsH9&0349>rnlf$z6(6c!zwHHML$*E7K@{qii0p z^omyRx4kYP5VX&u^WUo=oTJB(#r%r1q1Epz6HgUc{tU{T=Ni%`N(Cp z^)Oh-{CFf1c?X7#V-T{zS^Co2~0)GFukP zDvI9Yti?)OCr;gE*GBn?#ny>&qvw#^ z=G*M)@J*qoD_r|hBoKlhPnIz0WQ$0B>J46F!Qzsluf9{$Hj&n3O#|iT=Ts4nXg^V{ zzG=@U#BTQqiC0%C&nDC(t^_m=M$ZrRHk9m}VfI|OGYD3cil;;Hb@DJ`{`Pf(D?t<% z2j9M3*vFSabv`M~O2S3ft*-$@jK-3Km7jcJCQI!U8=P<1yO)95B9iDo3NvKLmrEyy z&p2L8kgd4zu*y+{9<*u48*lB@)yEBn!t@mXg1B#;jrzq8m@gQMB~f}5o8>;@#S+fS zjLh5pZ5tWm)K5T;(4aq^6<)?NO_Q;A$HAfO-rJ5>nU9 zAJfgxnAs5D}RN zi6ONhDWDA+LTcp8pz2@(L;#Mofcnwod^OBh0hk4(M5+dz8B|Xi)zue{A;1B+B^>F^ zM;LC$Pw*h(4~WWep?dPxq{TG@js+HHK5#6GKNCj!xUt|^^bgeA*@X(nVis4va4h!6 z>;!|#!*M@WOfNWA8^5?X4aa`x7e+GS*dMfhaO@94HkIK8#{v2AI-)Qb3c9VUCGzntUoawAP z0RZ6t0{}1>>{2lR28Ult0$>5c((nK*2EB|JOIRKNj&~Wq3*5{Xfx}`J)%+#~M}X+t zB|$W>)Hi_tm(zIsvdjQH0l&;Qo`hN^od6J)StOuH-=hG3dmsP-x4a>Nh+F2Hh~mFz z{`mE{03v$%CGbj@J{o|?Bl)eC$IlxiG=3Qg2}@e0nndD%Z2a+y7zJQZOG!|?8(faz z(SKYFuL%kTfXfu3P#EG;cPJDNw~QD?z%RWz6dFJ;!>|PWZ?(P*XEz#^AxtL2ktUw5 x1m3s&{jUK>T6%f;!hg)I+}Kn&FZl1jZ)b)ti$Qf2Mqx=Pg0QOU&V9Rt{|6q4GNu3k diff --git a/text/evaluation/details.tex b/text/evaluation/details.tex index 7ca5414..5ae5a58 100644 --- a/text/evaluation/details.tex +++ b/text/evaluation/details.tex @@ -1,103 +1,125 @@ -\section{Experimental Setting and Data Sets} +\section{Experimental setting and data sets} \label{sec:eval-dtl} - In this section we list all the relevant details regarding the evaluation setting (Section~\ref{subsec:eval-setup}), and we present the real and synthetic data sets that we used (Section~\ref{subsec:eval-dat}), along with the corresponding configurations (Section~\ref{subsec:eval-conf}). -\subsection{Machine Setup} +\subsection{Machine setup} \label{subsec:eval-setup} - -We implemented our experiments\footnote{Code available at \url{https://git.delkappa.com/manos/the-last-thing}} in Python $3$.$9$.$7$ and executed them on a machine with an Intel i$7$-$6700$HQ at $3$.$5$GHz CPU and $16$GB RAM, running Manjaro Linux $21$.$1$.$5$. -We repeated each experiment $100$ times and we report the mean over these iterations. \kat{It could be interesting to report also on the diagrams the std} +We implemented our experiments\footnote{Source code available at \url{https://git.delkappa.com/manos/the-last-thing}} in Python $3$.$9$.$7$ and executed them on a machine with an Intel i$7$-$6700$HQ at $3$.$5$GHz CPU and $16$GB RAM, running Manjaro Linux $21$.$1$.$5$. +We repeated each experiment $100$ times and we report the mean over these iterations. +% \kat{It could be interesting to report also on the diagrams the std} +% \mk{I'll keep it in mind.} \subsection{Data sets} \label{subsec:eval-dat} +We performed experiments on real (Section~\ref{subsec:eval-dat-real}) and synthetic data sets (Section~\ref{subsec:eval-dat-syn}). -\subsubsection{Real Data Sets} + +\subsubsection{Real data sets} +\label{subsec:eval-dat-real} +For uniformity and in order to be consistent, we sample from each of the following data sets the first $1,000$ entries that satisfy the configuration criteria that we discuss in detail in Section~\ref{subsec:eval-conf}. \paragraph{Copenhagen}~\cite{sapiezynski2019interaction} data set was collected via the smartphone devices of $851$ university students over a period of $4$ week as part of the Copenhagen Networks Study. Each device was configured to be discoverable by and to discover nearby Bluetooth devices every $5$ minutes. Upon discovery, each device registers (i)~the timestamp in seconds, (ii)~the device's unique identifier, (iii)~the unique identifier of the device that it discovered ($- 1$ when no device was found or $- 2$ for any non-participating device), and (iv)~the Received Signal Strength Indicator (RSSI) in dBm. Half of the devices have registered data at at least $81\%$ of the possible timestamps. -From this data set, we utilized the $1,000$ first contacts out of $12,167$ valid unique contacts of the device with identifier `$449$'. \kat{why only the 1000 first contacts? why device 449? why only one device and not multiple ones, and then report the mean?} +$3$ devices ($449$, $550$, $689$) satisfy our configuration criteria (Section~\ref{subsec:eval-conf}) within their first $1,000$ entries. +From those $3$ devices, we picked the first one, i.e.,~device with identifier `$449$', and utilized its $1,000$ first entries out of $12,167$ unique valid contacts. +% \kat{why only the 1000 first contacts? why device 449? why only one device and not multiple ones, and then report the mean?} +% \mk{I explained why 449 and I added a general explanation in the intro of the subsection.} \paragraph{HUE}~\cite{makonin2018hue} contains the hourly energy consumption data of $22$ residential customers of BCHydro, a provincial power utility in British Columbia. The measurements for each residence are saved individually and each measurement contains (i)~the date (YYYY-MM-DD), (ii)~the hour, and (iii)~the energy consumption in kWh. -In our experiments, we used the first $1,000$ out of $29,231$ measurements of the residence with identifier `$1$', average energy consumption equal to $0.88$kWh, and value range $[0.28$, $4.45]$. \kat{again, explain your choices. Moreover, you make some conclusions later on, based on the characteristics of the data set, for example the density of the measurement values. You should describe all these characteristics in these paragraphs.} +In our experiments, we used the first residence, i.e.,~residence with identifier `$1$', that satisfies our configuration criteria (Section~\ref{subsec:eval-conf}) within its first $1,000$ entries. +In those entries, out of a total of $29,231$ measurements, we estimated an average energy consumption equal to $0.88$kWh and a value range within $[0.28$, $4.45]$. +% \kat{again, explain your choices. Moreover, you make some conclusions later on, based on the characteristics of the data set, for example the density of the measurement values. You should describe all these characteristics in these paragraphs.} +% \mk{OK} \paragraph{T-drive}~\cite{yuan2010t} consists of $15$ million GPS data points of the trajectories of $10,357$ taxis in Beijing, spanning a period of $1$ week and a total distance of $9$ million kilometers. The taxis reported their location data on average every $177$ seconds and $623$ meters approximately. Each vehicle registers (i)~the taxi unique identifier, (ii)~the timestamp (YYYY-MM-DD HH:MM:SS), (iii)~longitude, and (iv)~latitude. These measurements are stored individually per vehicle. -We sampled the first $1000$ data items of the taxi with identifier `$2$'.\kat{again, explain your choices} +We sampled the first $1000$ data items of the taxi with identifier `$2$', which satisfied our configuration criteria (Section~\ref{subsec:eval-conf}). +% \kat{again, explain your choices} +% \mk{OK} + \subsubsection{Synthetic} +\label{subsec:eval-dat-syn} We generated synthetic time series of length equal to $100$ timestamps, for which we varied the number and distribution of {\thethings}. -In this way, we have a controlled data set that we can use to study the behaviour of our proposal. -\kat{more details needed. eg. what is the distributions and number of timestamps used? How many time series you generated? } -We take into account only the temporal order of the points and the position of regular and {\thething} events within the series. \kat{why is the value not important? at the energy consumption, they mattered} +In this way, we have a controlled data set that we can use to study the behavior of our proposal. +% \kat{more details needed. eg. what is the distributions and number of timestamps used? How many time series you generated? } +We take into account only the temporal order of the points and the position of regular and {\thething} events within the series. +In Section~\ref{subsec:eval-conf}, we explain in more detail our configuration criteria. +% \kat{why is the value not important? at the energy consumption, they mattered} \subsection{Configurations} \label{subsec:eval-conf} +% \kat{add some info here.. what are the configurations for? What does landmark percentage refer to, and how does it matter? } +We vary the {\thething} percentage (Section~\ref{subsec:eval-conf-lmdk}), i.e.,~the ratio of timestamps that we attribute to {\thethings} and regular events, in order to identify the limitations of our methodology. +For each data set, we implement a privacy mechanism that injects noise related to the type of its attribute values and we tune the parameters of each mechanism accordingly (Section~\ref{subsec:eval-conf-prv}). +Last but not least, we explain how we generate synthetic data sets with the desired degree of temporal correlation (Section~\ref{subsec:eval-conf-cor}). + -\kat{add some info here.. what are the configurations for? What does landmark percentage refer to, and how does it matter? } \subsubsection{{\Thething} percentage} - -In the Copenhagen data set, a landmark represents a time-stamp when a contact device is registered. -We achieve +\label{subsec:eval-conf-lmdk} +In the Copenhagen data set, a {\thething} represents a timestamp when a contact device is registered. +After identifying the unique contacts within the sample, we achieve each desired {\thethings} to regular events ratio by considering a list that contains a part of these contact devices. +In more detail, we achieve $0\%$ {\thethings} by considering an empty list of contact devices, $20\%$ by extending the list with $[3$, $6$, $11$, $12$, $25$, $29$, $36$, $39$, $41$, $46$, $47$, $50$, $52$, $56$, $57$, $61$, $63$, $78$, $80]$, $40\%$ with $[81$, $88$, $90$, $97$, $101$, $128$, $130$, $131$, $137$, $145$, $146$, $148$, $151$, $158$, $166$, $175$, $176]$, $60\%$ with $[181$, $182$, $192$, $195$, $196$, $201$, $203$, $207$, $221$, $230$, $235$, $237$, $239$, $241$, $254]$, $80\%$ with $[260$, $282$, $287$, $289$, $290$, $291$, $308$, $311$, $318$, $323$, $324$, $330$, $334$, $335$, $344$, $350$, $353$, $355$, $357$, $358$, $361$, $363]$, and $100\%$ by including all of the possible contacts. -\kat{How did you decide which devices to add at each point?} +% \kat{How did you decide which devices to add at each point?} +% \mk{I discussed it earlier.} -\kat{Say what time-stamps are landmarks in this data set. What is the consumption threshld?}In HUE, we get $0$\%, $20$\% $40$\%, $60$\%, $80$\%, and $100$\% {\thethings} by setting the energy consumption threshold below $0.28$kWh, $1.12$kWh, $0.88$kWh, $0.68$kWh, $0.54$kWh, $4.45$kWh respectively. +% \kat{Say what time-stamps are landmarks in this data set. What is the consumption threshld?} +% \mk{OK} +In HUE, we consider as {\thethings} the events that have energy consumption values below a certain threshold. +That is, we get $0$\%, $20$\% $40$\%, $60$\%, $80$\%, and $100$\% {\thethings} by setting the energy consumption threshold at $0.28$kWh, $1.12$kWh, $0.88$kWh, $0.68$kWh, $0.54$kWh, and $4.45$kWh respectively. -In T-drive, a landmark represents the time-stamp of a stay point. We achieved the desired {\thething} percentages by utilizing the method of Li et al.~\cite{li2008mining} for detecting stay points in trajectory data. +In T-drive, a {\thething} represents a location where a vehicle spend some time. +We achieved the desired {\thething} percentages by utilizing the method of Li et al.~\cite{li2008mining} for detecting stay points in trajectory data. In more detail, the algorithm checks for each data item if each subsequent item is within a given distance threshold $\Delta l$ and measures the time period $\Delta t$ between the present point and the last subsequent point. -We achieve $0$\%, $20$\% $40$\%, $60$\%, $80$\%, and $100$\% {\thethings} by setting the ($\Delta l$ in meters, $\Delta t$ in minutes) pairs input to the stay point discovery method as [($0$, $1000$), ($2095$, $30$), ($2790$, $30$), ($3590$, $30$), ($4825$, $30$), ($10350$, $30$)]. \kat{how did you come up with these numbers?} +After analyzing the data and experimenting with different pairs of distance and time period, we achieve $0$\%, $20$\% $40$\%, $60$\%, $80$\%, and $100$\% {\thethings} by setting the ($\Delta l$ in meters, $\Delta t$ in minutes) pairs input to the stay point discovery method as [($0$, $1000$), ($2095$, $30$), ($2790$, $30$), ($3590$, $30$), ($4825$, $30$), ($10350$, $30$)]. +% \kat{how did you come up with these numbers?} We generated synthetic data with \emph{skewed} (the {\thethings} are distributed towards the beginning/end of the series), \emph{symmetric} (in the middle), \emph{bimodal} (both end and beginning), and \emph{uniform} (all over the time series) {\thething} distributions. -In order to get {\thethings} with the above distribution features, we generate probability distributions with appropriate characteristics and sample from them, without replacement, the desired number of points. -%The generated distributions are representative of the cases that we wish to examine during the experiments. -For a left-skewed {\thething} distribution we would utilize a truncated distribution resulting from the restriction of the domain of a distribution to the beginning and end of the time series with its location shifted to the center of the right half of the series. -For a right-skewed .... -For a symmetric .. -For a bimodal .. -For uniform ... -\kat{repeat for all kinds of distributions} +In order to get {\thething} sets with the above distribution features, we generate probability distributions with restricted domain to the beginning and end of the time series, and sample from them, without replacement, the desired number of points. +For each case, we place the location, i.e.,~centre, of the distribution accordingly. +That is, for a symmetric we put the location in the middle of the time series and for a left/right skewed to the right/left. +For the bimodal we combine two mirrored skewed distributions. +Finally, for the uniform distribution we distribute the {\thethings} randomly throughout the time series. For consistency, we calculate the scale parameter depending on the length of the series by setting it equal to the series' length over a constant. -\kat{The following paragraph does not belong in this section..} -Notice that in our experiments, in the cases when we have $0\%$ and $100\%$ of the events being {\thethings}, we get the same behavior as in event- and user-level privacy respectively. -This happens due the fact that at each timestamp we take into account only the data items at the current timestamp and ignore the rest of the time series (event-level) when there are no {\thethings}. -Whereas, when each timestamp corresponds to a {\thething} we consider and protect all the events throughout the entire series (user-level). - - \subsubsection{Privacy parameters} - -\kat{Explain why you select each of these perturbation mechanisms for each of the datasets. Is the random response differential private? Mention it! } -To perturb the contact tracing data of the Copenhagen data set, we utilize the \emph{random response} technique~\cite{wang2017locally} to report with probability $p = \frac{e^\varepsilon}{e^\varepsilon + 1}$ whether the current contact is a {\thething} or not. +\label{subsec:eval-conf-prv} +% \kat{Explain why you select each of these perturbation mechanisms for each of the datasets. Is the random response differential private? Mention it! } +For all of te real data sets, we implement $\varepsilon$-differential privacy. +To perturb the contact tracing data of the Copenhagen data set, we utilize the \emph{random response} technique~\cite{wang2017locally}, and at each timestamp we report truthfully, with probability $p = \frac{e^\varepsilon}{e^\varepsilon + 1}$, whether the current contact is a {\thething} or not. We randomize the energy consumption in HUE with the Laplace mechanism (described in detail in Section~\ref{subsec:prv-mech}). -We inject noise to the spatial values in T-drive that we sample from the Planar Laplace mechanism~\cite{andres2013geo}. +For T-drive, we perturb the location data wit noise that we sample from the Planar Laplace mechanism~\cite{andres2013geo}. -We set the privacy budget $\varepsilon = 1$, and, for simplicity, we assume that for every query sensitivity it holds that $\Delta f = 1$. \kat{why don't you consider other values as well?} -For the experiments performed on the synthetic data sets, the original values to be released do not influence the outcome of our conclusions, thus we ignore them. -\kat{why are the values not important for the synthetic dataset? This seems a little weird, when said out of context.. our goal is to perturb the values, but do not really care about the way we perturb our values?} +We set the privacy budget $\varepsilon = 1$ for all of our experiments and, for simplicity, we assume that for every query sensitivity it holds that $\Delta f = 1$. +% \kat{why don't you consider other values as well?} +For the experiments performed on the synthetic data sets, the original values to be released do not influence the outcome of our conclusions, thus we ignore them. +% \kat{why are the values not important for the synthetic dataset? This seems a little weird, when said out of context.. our goal is to perturb the values, but do not really care about the way we perturb our values?} % Finally, notice that, depending on the results' variation, most diagrams are in logarithmic scale. \subsubsection{Temporal correlation} - -\kat{Did you find any correlation in the other data? Do you need the correlation matrix to be known a priori? Describe a little why you did not use the real data for correlations } +\label{subsec:eval-conf-cor} +% \kat{Did you find any correlation in the other data? Do you need the correlation matrix to be known a priori? Describe a little why you did not use the real data for correlations } +Despite the inherent presence of temporal correlation in time series, it is challenging to correctly discover and quantify it. +For this reason, and in order to create a more controlled environment for our experiments, we chose to conduct tests relevant to temporal correlation using synthetic data sets. We model the temporal correlation in the synthetic data as a \emph{stochastic matrix} $P$, using a \emph{Markov Chain}~\cite{gagniuc2017markov}. $P$ is a $n \times n$ matrix, where the element $P_{ij}$ %at the $i$th row of the $j$th column that diff --git a/text/evaluation/main.tex b/text/evaluation/main.tex index 2a9989c..fae6cd1 100644 --- a/text/evaluation/main.tex +++ b/text/evaluation/main.tex @@ -1,7 +1,6 @@ \chapter{Evaluation} \label{ch:eval} - -In this chapter we present the experiments that we performed in order to evaluate {\thething} Privacy (Chapter~\ref{ch:lmdk-prv}) on real and synthetic data sets. +In this chapter we present the experiments that we performed in order to evaluate {\thething} privacy (Chapter~\ref{ch:lmdk-prv}) on real and synthetic data sets. Section~\ref{sec:eval-dtl} contains all the details regarding the data sets the we used for our experiments along with the system configurations. Section~\ref{sec:eval-lmdk} evaluates the data utility of the {\thething} privacy mechanisms that we designed in Section~\ref{sec:thething} and investigates the behavior of the privacy loss under temporal correlation for different distributions of {\thethings}. Section~\ref{sec:eval-lmdk-sel} justifies our decisions while designing the privacy-preserving {\thething} selection component in Section~\ref{sec:theotherthing} and the data utility impact of the latter. diff --git a/text/evaluation/summary.tex b/text/evaluation/summary.tex index c4470b2..e343115 100644 --- a/text/evaluation/summary.tex +++ b/text/evaluation/summary.tex @@ -1,9 +1,10 @@ \section{Summary} \label{sec:eval-sum} - In this chapter we presented the experimental evaluation of the {\thething} privacy mechanisms and the privacy-preserving {\thething} selection mechanism that we developed in Chapter~\ref{ch:lmdk-prv}, on real and synthetic data sets. -The Adaptive mechanism is the most reliable and best performing mechanism, in terms of overall data utility, with minimal tuning across most cases. +The Adaptive mechanism is the most reliable and best performing mechanism, in terms of overall data utility, with minimal tuning across most of the cases. Skip performs optimally in data sets with a smaller target value range, where approximation fits best. -The {\thething} selection component introduces a reasonable data utility decline to all of our mechanisms however, the Adaptive handles it well and bounds the data utility to higher levels compared to user-level protection.\kat{it would be nice to see it clearly on Figure 5.5. (eg, by including another bar that shows adaptive without landmark selection)} +The {\thething} selection mechanism introduces a reasonable data utility decline to all of our mechanisms however, the Adaptive handles it well and bounds the data utility to higher levels compared to user-level protection. +% \kat{it would be nice to see it clearly on Figure 5.5. (eg, by including another bar that shows adaptive without landmark selection)} +% \mk{Done.} In terms of temporal correlation, we observe that under moderate and strong temporal correlation, a greater average regular--{\thething} event distance in a {\thething} distribution causes greater overall privacy loss. -Finally, the contribution of the {\thething} privacy on enhancing the data quality, while preserving $\epsilon$-differential privacy is demonstrated by the fact that the selected, Adaptive mechanism provides better data quality than the user-level mechanism. +Finally, the contribution of the {\thething} privacy on enhancing the data utility, while preserving $\epsilon$-differential privacy, is demonstrated by the fact that the selected Adaptive mechanism provides better data utility than the user-level mechanism. diff --git a/text/evaluation/theotherthing.tex b/text/evaluation/theotherthing.tex index e82136b..b21c15c 100644 --- a/text/evaluation/theotherthing.tex +++ b/text/evaluation/theotherthing.tex @@ -1,25 +1,26 @@ \section{Selection of landmarks} \label{sec:eval-lmdk-sel} - -In this section, we present the experiments on the methodology for the {\thethings} selection presented in Section~\ref{subsec:lmdk-sel-sol}, on the real and the synthetic data sets. -With the experiments on the synthetic data sets (Section~\ref{subsec:sel-utl}) we show the normalized Euclidean and Wasserstein distances \kat{is this distance the landmark distance that we saw just before ? clarify } of the time series histograms for various distributions and {\thething} percentages. +In this section, we present the experiments on the methodology for the {\thethings} selection presented in Section~\ref{subsec:lmdk-sel-sol}, on the real and synthetic data sets. +With the experiments on the synthetic data sets (Section~\ref{subsec:sel-utl}) we show the normalized Euclidean and Wasserstein distance metrics (not to be confused with the temporal distances in Figure~\ref{fig:avg-dist}) +% \kat{is this distance the landmark distance that we saw just before ? clarify } +of the time series histograms for various distributions and {\thething} percentages. This allows us to justify our design decisions for our concept that we showcased in Section~\ref{subsec:lmdk-sel-sol}. -With the experiments on the real data sets (Section~\ref{subsec:sel-prv}), we show the performance in terms of utility of our three {\thething} mechanisms in combination with the privacy preserving {\thething} selection component. -\kat{Mention whether it improves the original proposal or not.} +With the experiments on the real data sets (Section~\ref{subsec:sel-prv}), we show the performance in terms of utility of our three {\thething} mechanisms in combination with the privacy-preserving {\thething} selection mechanism, which enhances the privacy protection of our concept. +% \kat{Mention whether it improves the original proposal or not.} \subsection{{\Thething} selection utility metrics} \label{subsec:sel-utl} - Figure~\ref{fig:sel-dist} demonstrates the normalized distance that we obtain when we utilize either (a)~the Euclidean or (b)~the Wasserstein distance metric to obtain a set of {\thethings} including regular events. \begin{figure}[htp] \centering \subcaptionbox{Euclidean\label{fig:sel-dist-norm}}{% - \includegraphics[width=.5\linewidth]{evaluation/sel-dist-norm}% + \includegraphics[width=.49\linewidth]{evaluation/sel-dist-norm}% }% + \hfill \subcaptionbox{Wasserstein\label{fig:sel-dist-emd}}{% - \includegraphics[width=.5\linewidth]{evaluation/sel-dist-emd}% + \includegraphics[width=.49\linewidth]{evaluation/sel-dist-emd}% }% \caption{The normalized (a)~Euclidean, and (b)~Wasserstein distance of the generated {\thething} sets for different {\thething} percentages.} \label{fig:sel-dist} @@ -29,67 +30,71 @@ Comparing the results of the Euclidean distance in Figure~\ref{fig:sel-dist-norm % (1 + (0.25 + 0.25 + 0.45 + 0.45)/4 + (0.25 + 0.25 + 0.3 + 0.3)/4 + (0.2 + 0.2 + 0.2 + 0.2)/4 + (0.15 + 0.15 + 0.15 + 0.15)/4)/6 % (1 + (0.1 + 0.1 + 0.25 + 0.25)/4 + (0.075 + 0.075 + .15 + 0.15)/4 + (0.075 + 0.075 + 0.1 + 0.1)/4 + (0.025 + 0.025 + 0.025 + 0.025)/4)/6 The maximum difference per {\thething} percentage is approximately $0.2$ for the former and $0.15$ for the latter between the bimodal and skewed {\thething} distributions. -Overall, the Euclidean achieves a mean normalized distance of $0.3$ and the Wasserstein $0.2$. -Therefore, and by observing Figure~\ref{fig:sel-dist}, the Wasserstein distance demonstrates a less consistent performance and less linear behavior among all possible {\thething} distributions. -Thus, we choose to utilize the Euclidean distance metric for the implementation of the privacy-preserving {\thething} selection in Section~\ref{subsec:lmdk-sel-sol}. +Overall, the Euclidean distance achieves a mean normalized distance of $0.3$ while the Wasserstein distance a mean normalized distance that is equal to $0.2$. +Therefore, and by observing Figure~\ref{fig:sel-dist}, Wasserstein demonstrates a less consistent performance and less linear behavior among all possible {\thething} distributions. +Thus, we choose to utilize the Euclidean distance metric for the implementation of the privacy-preserving {\thething} selection mechanism in Section~\ref{subsec:lmdk-sel-sol}. \subsection{Privacy budget tuning} \label{subsec:sel-eps} - -In Figure~\ref{fig:sel-eps} we test the Uniform model in real data by investing different ratios ($1$\%, $10$\%, $25$\%, and $50$\%) of the available privacy budget $\varepsilon$ in the {\thething} selection mechanism, in order to figure out the optimal ratio value. +In Figure~\ref{fig:sel-eps} we test the Uniform mechanism in real data by investing different ratios ($1$\%, $10$\%, $25$\%, and $50$\%) of the available privacy budget $\varepsilon$ in the {\thething} selection mechanism and the remaining to perturbing the data values, in order to figure out the optimal ratio value. Uniform is our baseline implementation, and hence allows us to derive more accurate conclusions in this case. -In general, greater ratios will result in more accurate, i.e.,~smaller, {\thething} sets and less accurate values in the released data sets. +In general, we are expecting to observe that greater ratios will result in more accurate, i.e.,~smaller, {\thething} sets and less accurate values in the released data. \begin{figure}[htp] \centering \subcaptionbox{Copenhagen\label{fig:copenhagen-sel-eps}}{% - \includegraphics[width=.5\linewidth]{evaluation/copenhagen-sel-eps}% + \includegraphics[width=.49\linewidth]{evaluation/copenhagen-sel-eps}% }% \hspace{\fill} \\ \bigskip \subcaptionbox{HUE\label{fig:hue-sel-eps}}{% - \includegraphics[width=.5\linewidth]{evaluation/hue-sel-eps}% + \includegraphics[width=.49\linewidth]{evaluation/hue-sel-eps}% }% + \hfill \subcaptionbox{T-drive\label{fig:t-drive-sel-eps}}{% - \includegraphics[width=.5\linewidth]{evaluation/t-drive-sel-eps}% + \includegraphics[width=.49\linewidth]{evaluation/t-drive-sel-eps}% }% - \caption{The mean absolute error (a)~as a percentage, (b)~in kWh, and (c)~in meters of the released data for different {\thething} percentages. We apply the Uniform {\thething} privacy model and vary the ratio of the privacy budget $\varepsilon$ that we allocate to the {\thething} selection component.} + \caption{The mean absolute error (a)~as a percentage, (b)~in kWh, and (c)~in meters of the released data for different {\thething} percentages. We apply the Uniform {\thething} privacy mechanism and vary the ratio of the privacy budget $\varepsilon$ that we allocate to the {\thething} selection mechanism.} \label{fig:sel-eps} \end{figure} -The application of the randomized response mechanism, in the Copenhagen data set, is tolerant to the fluctuations of the privacy budget. -For HUE and T-drive, we observe that our implementation performs better for lower ratios, e.g.,~$0.01$, where we end up allocating the majority of the available privacy budget to the data release process instead of the {\thething} selection component. -The results of this experiment indicate that we can safely allocate the majority of $\varepsilon$ for publishing the data values, and therefore achieve better data utility, while providing more robust privacy protection to the {\thething} timestamp set. +The application of the randomized response mechanism, in the Copenhagen data set, is tolerant to the fluctuations of the privacy budget and maintains a relatively constant performance in terms of data utility. +For HUE and T-drive, we observe that our implementation performs better for lower ratios, e.g.,~$0.01$, where we end up allocating the majority of the available privacy budget to the data release process instead of the {\thething} selection mechanism. +The results of this experiment indicate that we can safely allocate the majority of $\varepsilon$ for publishing the data values, and therefore achieve better data utility, while providing more robust privacy protection to the {\thething} set. \subsection{Budget allocation and {\thething} selection} \label{subsec:sel-prv} - -Figure~\ref{fig:real-sel} exhibits the performance of Skip, Uniform, and Adaptive (see Section~\ref{subsec:lmdk-mechs}) in combination with the {\thething} selection component. +Figure~\ref{fig:real-sel} exhibits the performance of Skip, Uniform, and Adaptive mechanisms (presented in detail in Section~\ref{subsec:lmdk-mechs}) in combination with the {\thething} selection mechanism (Section~\ref{subsec:lmdk-sel-sol}). \begin{figure}[htp] \centering \subcaptionbox{Copenhagen\label{fig:copenhagen-sel}}{% - \includegraphics[width=.5\linewidth]{evaluation/copenhagen-sel}% + \includegraphics[width=.49\linewidth]{evaluation/copenhagen-sel}% }% - \hspace{\fill} + \hfill \\ \bigskip \subcaptionbox{HUE\label{fig:hue-sel}}{% - \includegraphics[width=.5\linewidth]{evaluation/hue-sel}% + \includegraphics[width=.49\linewidth]{evaluation/hue-sel}% }% + \hfill \subcaptionbox{T-drive\label{fig:t-drive-sel}}{% - \includegraphics[width=.5\linewidth]{evaluation/t-drive-sel}% + \includegraphics[width=.49\linewidth]{evaluation/t-drive-sel}% }% - \caption{The mean absolute error (a)~as a percentage, (b)~in kWh, and (c)~in meters of the released data for different {\thething} percentages.} + \caption{ + The mean absolute error (a)~as a percentage, (b)~in kWh, and (c)~in meters of the released data, for different {\thething} percentages, with the incorporation of the privacy-preserving {\thething} selection mechanism. + The light short horizontal lines indicate the corresponding measurements from Figure~\ref{fig:real} without the {\thething} selection mechanism. + } \label{fig:real-sel} \end{figure} -In comparison with the utility performance without the {\thething} selection component (Figure~\ref{fig:real}), we notice a slight deterioration for all three models. -This is natural since we allocated part of the available privacy budget to the privacy-preserving {\thething} selection component, which in turn increased the number of {\thethings}. -Therefore, there is less privacy budget available for data publishing throughout the time series for $0$\% and $100$\% {\thethings}. -\kat{why not for the other percentages?} -Skip performs best in our experiments with HUE, due to the low range in the energy consumption and the high scale of the Laplace noise, which it avoids due to the employed approximation. -However, for the Copenhagen data set and T-drive Skip attains greater mean absolute error than the user-level protection scheme, which exposes no benefit w.r.t. user-level protection. -Overall, Adaptive has a consistent performance in terms of utility for all of the data sets that we experimented with, and always outperforms the user-level privacy. +In comparison with the utility performance without the {\thething} selection mechanism (light short horizontal lines), we notice a slight deterioration for all three mechanisms. +This is natural since we allocated part of the available privacy budget to the privacy-preserving {\thething} selection mechanism, which in turn increased the number of {\thethings}, except for the case of $100$\% {\thethings}. +Therefore, there is less privacy budget available for data publishing throughout the time series. +% for $0$\% and $100$\% {\thethings}. +% \kat{why not for the other percentages?} +Skip performs best in our experiments with HUE, due to the low range in the energy consumption and the high scale of the Laplace noise that it avoids due to the employed approximation. +However, for the Copenhagen data set and T-drive, Skip attains greater mean absolute error than the user-level protection scheme, which exposes no benefit with respect to user-level protection. +Overall, Adaptive has a consistent performance in terms of utility for all of the data sets that we experimented with, and almost always outperforms the user-level privacy. Thus, it is selected as the best mechanism to use in general. diff --git a/text/evaluation/thething.tex b/text/evaluation/thething.tex index abd6f70..2bbb11f 100644 --- a/text/evaluation/thething.tex +++ b/text/evaluation/thething.tex @@ -1,66 +1,90 @@ -\section{Landmark events} +\section{{\Thething} events} \label{sec:eval-lmdk} - % \kat{After discussing with Dimitris, I thought you are keeping one chapter for the proposals of the thesis. In this case, it would be more clean to keep the theoretical contributions in one chapter and the evaluation in a separate chapter. } % \mk{OK.} In this section, we present the experiments that we performed, to test the methodology that we presented in Section~\ref{subsec:lmdk-sol}, on real and synthetic data sets. -With the experiments on the real data sets (Section~\ref{subsec:lmdk-expt-bgt}), we show the performance in terms of data utility of our three {\thething} privacy budget allocation schemes: Skip, Uniform and Adaptive. -We define data utility as the Mean Absolute Error introduced by the privacy mechanism. -We compare with the event and user differential privacy, and show that in the general case, {\thething} privacy allows for better data utility than user differential privacy. - -With the experiments on the synthetic data sets (Section~\ref{subsec:lmdk-expt-cor}) we show the privacy loss \kat{in the previous set of experiments we were measuring the MAE, now we are measuring the privacy loss... Why is that? Isn't it two sides of the same coin? }by our framework when tuning the size and statistical characteristics of the input {\thething} set $L$ with special emphasis on how the privacy loss under temporal correlation is affected by the number and distribution of the {\thethings}. -\kat{mention briefly what you observe} +With the experiments on the real data sets (Section~\ref{subsec:lmdk-expt-bgt}), we show the performance in terms of data utility of our three {\thething} privacy mechanisms: Skip, Uniform and Adaptive. +We define data utility as the mean absolute error introduced by the privacy mechanism. +We compare with the event- and user-level differential privacy protection levels, and show that, in the general case, {\thething} privacy allows for better data utility than user-level differential privacy while balancing between the two protection levels. +With the experiments on the synthetic data sets (Section~\ref{subsec:lmdk-expt-cor}) we show the overall privacy loss, +% \kat{in the previous set of experiments we were measuring the MAE, now we are measuring the privacy loss... Why is that? Isn't it two sides of the same coin? } +i.e.,~the privacy budget $\varepsilon$, under temporal correlation within our framework when tuning the size and statistical characteristics of the input {\thething} set $L$. +% \kat{mention briefly what you observe} +We observe that a greater average {\thething}--regular event distance in a time series can result into greater overall privacy loss under moderate and strong temporal correlation. \subsection{Budget allocation schemes} \label{subsec:lmdk-expt-bgt} Figure~\ref{fig:real} exhibits the performance of the three mechanisms, Skip, Uniform, and Adaptive applied on the three data sets that we study. +Notice that, in the cases when we have $0\%$ and $100\%$ of the events being {\thethings}, we get the same behavior as in event- and user-level privacy respectively. +This happens due the fact that at each timestamp we take into account only the data items at the current timestamp and ignore the rest of the time series (event-level) when there are no {\thethings}. +Whereas, when each timestamp corresponds to a {\thething} we consider and protect all the events throughout the entire series (user-level). % For the Geolife data set (Figure~\ref{fig:geolife}), Skip has the best performance (measured in Mean Absolute Error, in meters) because it invests the most budget overall at every regular event, by approximating the {\thething} data based on previous releases. % Due to the data set's high density (every $1$--$5$ seconds or every $5$--$10$ meters per point) approximating constantly has a low impact on the data utility. % On the contrary, the lower density of the T-drive data set (Figure~\ref{fig:t-drive}) has a negative impact on the performance of Skip. -For the Copenhagen data set (Figure~\ref{fig:copenhagen}), Adaptive has a constant\kat{it is not constant, for 0 it is much lower} overall performance and performs best for $0$\%, $60$\%, and $80$\% {\thethings} \kat{this is contradictory: you say that it is constant overall, and then that it is better for certain percentages. }. -We notice that for $0$\% {\thethings}, it achieves better utility than the event-level protection.\kat{what does this mean? how is it possible?} -The Skip model excels, compared to the others, at cases where it needs to approximate $20$\%--$40$\% or $100$\% of the times.\kat{it seems a little random.. do you have an explanation? (rather few times or all?)} - -The combination of the small range of measurements in HUE ($[0.28$, $4.45]$ with an average of $0.88$kWh) and the large scale in the Laplace mechanism, results in a low mean absolute error for Skip (Figure~\ref{fig:hue}). -In general, a scheme that favors approximation over noise injection would achieve a better performance in this case. -\kat{why?explain} -However, the Adaptive model performs by far better than Uniform and strikes a nice balance\kat{???} between event- and user-level protection for all {\thething} percentages. - -In the T-drive data set (Figure~\ref{fig:t-drive}), the Adaptive mechanism outperforms Uniform by $10$\%--$20$\% for all {\thething} percentages greater than $40$\% and Skip by more than $20$\%. -The lower density (average distance of $623$m) of the T-drive data set has a negative impact on the performance of Skip; republishing a previous perturbed value is now less accurate than perturbing the new location. - - \begin{figure}[htp] \centering \subcaptionbox{Copenhagen\label{fig:copenhagen}}{% - \includegraphics[width=.5\linewidth]{evaluation/copenhagen}% + \includegraphics[width=.49\linewidth]{evaluation/copenhagen}% }% - \hspace{\fill} \\ \bigskip \subcaptionbox{HUE\label{fig:hue}}{% - \includegraphics[width=.5\linewidth]{evaluation/hue}% + \includegraphics[width=.49\linewidth]{evaluation/hue}% }% + \hfill \subcaptionbox{T-drive\label{fig:t-drive}}{% - \includegraphics[width=.5\linewidth]{evaluation/t-drive}% + \includegraphics[width=.49\linewidth]{evaluation/t-drive}% }% \caption{The mean absolute error (a)~as a percentage, (b)~in kWh, and (c)~in meters of the released data for different {\thething} percentages.} \label{fig:real} \end{figure} -In general, we can claim that the Adaptive is the most reliable and best performing mechanism with minimal tuning\kat{what does minimal tuning mean?}, if we take into consideration the drawbacks of the Skip mechanism mentioned in Section~\ref{subsec:lmdk-mechs}. \kat{you can mention them also here briefly, and give the pointer for the section} -Moreover, designing a data-dependent sampling scheme \kat{what would be the main characteristic of the scheme? that it picks landmarks how?} would possibly\kat{possibly is not good enough, if you are sure remove it. Otherwise mention that more experiments need to be done?} result in better results for Adaptive. +For the Copenhagen data set (Figure~\ref{fig:copenhagen}), Adaptive has an +% constant +% \kat{it is not constant, for 0 it is much lower} +overall consistent performance and works best for $60$\% and $80$\% {\thethings}. +% \kat{this is contradictory: you say that it is constant overall, and then that it is better for certain percentages. }. +% \mk{`Consistent' is the right word.} +We notice that for $0$\% {\thethings}, it achieves better utility than the event-level protection +% \kat{what does this mean? how is it possible?} +due to the combination of more available privacy budget per timestamp (due to the absence of {\thethings}) and its adaptive sampling methodology. +The Skip model excels, compared to the others, at cases where it needs to approximate $20$\%, $40$\%, or $100$\% of the times. +% \kat{it seems a little random.. do you have an explanation? (rather few times or all?)} +In general, we notice that, for this data set, it is more beneficial to either invest more privacy budget per event or prefer approximation over introducing randomization. + +The combination of the small range of measurements in HUE ($[0.28$, $4.45]$ with an average of $0.88$kWh) and the large scale in the Laplace mechanism, allows for schemes that favor approximation over noise injection to achieve a better performance in terms of data utility. +Hence, Skip (Figure~\ref{fig:hue}) achieves a constant low mean absolute error. +% \kat{why?explain} +Regardless, the Adaptive mechanism performs by far better than Uniform and +% strikes a nice balance\kat{???} +balances between event- and user-level protection for all {\thething} percentages. + +In the T-drive data set (Figure~\ref{fig:t-drive}), the Adaptive mechanism outperforms Uniform by $10$\%--$20$\% for all {\thething} percentages greater than $40$\% and Skip by more than $20$\%. +The lower density (average distance of $623$m) of the T-drive data set has a negative impact on the performance of Skip; republishing a previous perturbed value is now less accurate than perturbing the new location. + +Principally, we can claim that the Adaptive is the most reliable and best performing mechanism, +% with a minimal and generic parameter tuning +% \kat{what does minimal tuning mean?} +if we take into consideration the drawbacks of the Skip mechanism, particularly in spatiotemporal data, e.g., sporadic location data publishing~\cite{gambs2010show, russell2018fitness} or misapplying location cloaking~\cite{xssfopes2020tweet}, that could lead to the indication of privacy-sensitive attribute values. +% (mentioned in Section~\ref{subsec:lmdk-mechs}) +% \kat{you can mention them also here briefly, and give the pointer for the section} +Moreover, implementing a more advanced and data-dependent sampling method +% \kat{what would be the main characteristic of the scheme? that it picks landmarks how?} +that accounts for changes in the trends of the input data and adapts its rate accordingly, would +% possibly +% \kat{possibly is not good enough, if you are sure remove it. Otherwise mention that more experiments need to be done?} +result in a more effective budget allocation that would improve the performance of Adaptive in terms of data utility. \subsection{Temporal distance and correlation} \label{subsec:lmdk-expt-cor} -As previously mentioned, temporal correlations are inherent in continuous publishing, and they are the cause of supplementary privacy leakage in the case of privacy preserving data publication. -In this section, we are interested in studying the effect that the distance of the {\thethings} from every event have on the leakage caused by temporal correlations. +As previously mentioned, temporal correlation is inherent in continuous publishing, and they are the cause of supplementary privacy loss in the case of privacy-preserving data publication. +In this section, we are interested in studying the effect that the distance of the {\thethings} from every event have on the loss caused by temporal correlation. Figure~\ref{fig:avg-dist} shows a comparison of the average temporal distance of the events from the previous/next {\thething} or the start/end of the time series for various distributions in our synthetic data. More specifically, we model the distance of an event as the count of the total number of events between itself and the nearest {\thething} or the series edge. @@ -68,7 +92,7 @@ More specifically, we model the distance of an event as the count of the total n \begin{figure}[htp] \centering \includegraphics[width=.5\linewidth]{evaluation/avg-dist}% - \caption{Average temporal distance of the events from the {\thethings} for different {\thethings} percentages within a time series in various {\thethings} distributions.} + \caption{Average temporal distance of regular events from the {\thethings} for different {\thethings} percentages within a time series in various {\thething} distributions.} \label{fig:avg-dist} \end{figure} @@ -79,28 +103,39 @@ On the contrary, distributing the {\thethings} at one part of the sequence, as i This study provides us with different distance settings that we are going to use in the subsequent temporal leakage study. Figure~\ref{fig:dist-cor} illustrates a comparison among the aforementioned distributions regarding the overall privacy loss under (a)~weak, (b)~moderate, and (c)~strong temporal correlation degrees. -The line shows the overall privacy loss---for all cases of {\thethings} distribution---without temporal correlation. +The line shows the overall privacy loss---for all cases of {\thething} distribution---without temporal correlation. \begin{figure}[htp] \centering \subcaptionbox{Weak correlation\label{fig:dist-cor-wk}}{% - \includegraphics[width=.5\linewidth]{evaluation/dist-cor-wk}% + \includegraphics[width=.49\linewidth]{evaluation/dist-cor-wk}% }% - \hspace{\fill} + \hfill \\ \bigskip \subcaptionbox{Moderate correlation\label{fig:dist-cor-mod}}{% - \includegraphics[width=.5\linewidth]{evaluation/dist-cor-mod}% + \includegraphics[width=.49\linewidth]{evaluation/dist-cor-mod}% }% + \hfill \subcaptionbox{Strong correlation\label{fig:dist-cor-stg}}{% - \includegraphics[width=.5\linewidth]{evaluation/dist-cor-stg}% + \includegraphics[width=.49\linewidth]{evaluation/dist-cor-stg}% }% - \caption{Privacy loss \kat{what is the unit for privacy loss? I t should appear on the diagram} for different {\thethings} percentages and distributions under (a)~weak, (b)~moderate, and (c)~strong degrees of temporal correlation. - The line shows the overall privacy loss without temporal correlation.} + \caption{ + The overall privacy loss (privacy budget $\varepsilon$) + % \kat{what is the unit for privacy loss? I t should appear on the diagram} + % \mk{It's the privacy budget epsilon} + for different {\thething} percentages and distributions under (a)~weak, (b)~moderate, and (c)~strong degrees of temporal correlation. + The line shows the overall privacy loss without temporal correlation. + } \label{fig:dist-cor} \end{figure} -In combination with Figure~\ref{fig:avg-dist}, we conclude that a greater average event--{\thething} event \kat{it was even, I changed it to event but do not know what youo want ot say} distance in a distribution can result into greater overall privacy loss under moderate and strong temporal correlation. +In combination with Figure~\ref{fig:avg-dist}, we conclude that a greater average {\thething}--regular event +% \kat{it was even, I changed it to event but do not know what youo want ot say} +% \mk{Fixed} +distance in a distribution can result into greater overall privacy loss under moderate and strong temporal correlation. This is due to the fact that the backward/forward privacy loss accumulates more over time in wider spaces without {\thethings} (see Section~\ref{sec:correlation}). Furthermore, the behavior of the privacy loss is as expected regarding the temporal correlation degree: a stronger correlation degree generates higher privacy loss while widening the gap between the different distribution cases. -On the contrary, a weaker correlation degree makes it harder to differentiate among the {\thethings} distributions. -The privacy loss under a weak correlation degree converge \kat{with what?}. +On the contrary, a weaker correlation degree makes it harder to differentiate among the {\thething} distributions. +The privacy loss under a weak correlation degree converge +% \kat{with what?} +with all possible distributions for all {\thething} percentages. From 0f0e64eac184be26d4025c64d49ab3fe748764ae Mon Sep 17 00:00:00 2001 From: Manos Date: Thu, 14 Oct 2021 06:12:48 +0200 Subject: [PATCH 02/23] Minor corrections --- code/expt/dist_cor.py | 24 ++++++++++++------------ graphics/evaluation/dist-cor-stg.pdf | Bin 16037 -> 16568 bytes text/related/main.tex | 2 +- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/code/expt/dist_cor.py b/code/expt/dist_cor.py index 039c5db..9b37d4b 100644 --- a/code/expt/dist_cor.py +++ b/code/expt/dist_cor.py @@ -45,7 +45,7 @@ def main(args): x_margin = bar_width*(len(dist_type)/2 + 1) plt.xlim(x_i.min() - x_margin, x_i.max() + x_margin) # The y axis - plt.ylabel('Privacy loss') # Set y axis label. + plt.ylabel('Overall privacy loss') # Set y axis label. plt.yscale('log') plt.ylim(epsilon/10, 100*len(seq)) # plt.ylim(0, 10000) @@ -55,15 +55,15 @@ def main(args): e = np.zeros(len(lmdk_n)) a = np.zeros(len(lmdk_n)) for i, n in enumerate(lmdk_n): - for r in range(args.reps): + for r in range(args.iter): # Generate landmarks lmdks = lmdk_lib.get_lmdks(seq, n, d) # Uniform budget allocation e_cur = lmdk_bgt.uniform(seq, lmdks, epsilon) _, _, a_cur = gdp.tpl_lmdk_mem(e_cur, p, p, seq, lmdks) # Save privacy loss - e[i] += np.sum(e_cur)/args.reps - a[i] += np.sum(a_cur)/args.reps + e[i] += np.sum(e_cur)/args.iter + a[i] += np.sum(a_cur)/args.iter # Set label label = lmdk_lib.dist_type_to_str(d_i) if d_i == 1: @@ -94,21 +94,21 @@ def main(args): print(' [OK]', flush=True) -''' - Parse arguments. - - Optional: - reps - The number of repetitions. - time - The time limit of the sequence. -''' def parse_args(): + ''' + Parse arguments. + + Optional: + iter - The number of repetitions. + time - The time limit of the sequence. + ''' # Create argument parser. parser = argparse.ArgumentParser() # Mandatory arguments. # Optional arguments. - parser.add_argument('-r', '--reps', help='The number of repetitions.', type=int, default=1) + parser.add_argument('-i', '--iter', help='The number of repetitions.', type=int, default=1) parser.add_argument('-t', '--time', help='The time limit of the sequence.', type=int, default=100) # Parse arguments. diff --git a/graphics/evaluation/dist-cor-stg.pdf b/graphics/evaluation/dist-cor-stg.pdf index 8df46a5838cdc07418aa6b60ab54bb3007fc7a90..4d6ab406a34346663c30884c0589a1d9cea78970 100644 GIT binary patch delta 4407 zcmZuyc_5T~7j6hQTtq4rMj6I7`rDGL0#t(46$L0LYbrG?_LuLO}hVGkh0#eqYnGgXb8=bu|SbDGC9-FCbj zurXB_3S1}D>S4r;amSFEv4b3?Tk9PXH~()m4lBKUscu^H#`&AhpL7ac6I|AZ-t%o0 zpIPqDt#@#mE_E3xasiV>Hb#0^Djc)$f0!%dvYW5JffVEfJT&=mTev^0nB6wABiTEc z|ERhL!-Nn{f_y|Fv?q1uBZ&Xm*qu|wnRi^h_djoS-{k48v{A!1{&k#H%)fmj)Pv)|)x z_r(O(GmdCC(bTh4qR%#`N4{P-TSM{T&w~Bt&2Peb z%C$BuSky{{)JauR5^JhOhH)sJcDf9|!EKkr@_!IV3x`6^&>G!~ghAucoXVzcC`1qI9rfzBIQ zf;$qY+uzuX#9Td(xtvZf9v~xXHF@|?3PJ|vZ+Sswb zfgRBenNZQG3O8$C>Q!ddD(kV#g6~ehasPnE)rcX@J92>rY>#u+R7^#`48U@>CUmkYp#Ekt2{L$ zcJ1=*;Tp%zc*`%{BdOzy^(=e&QG2_uZ=*VFK3OUB^Dm9I%fp>U$H7a2ia=mDM=#r& z{Xu6qCG5EB{a&^CbEOL;vbTS&iqh3Zr(BorA#|HTT6AUERD?=MdF+*~hcXn-5lhQD zdh=&5u#*N8tY@1d6ZFN1fo>Bwg5E0q785TpmpFVV&7e=n-XUbs!9IE5akKv{?R{7cOP+Y*xoZ+Y!tm- z^-On~tYGkRa^Q(pm|D`MCm%@I;0)m5m2)pLNSrZGTC#E2i{XaM^fc|}>48E1OMh^4{%vpmE;|Sqj{7MD|4r~ZL!UIO1=jx zQWb4{^zhTUWNrNn2`wquK~M9ozO||!UmWUeVCwkT_&OYjI&DmeDDrxub>9F7XjCO5 zM#XFmrV2OE2!_r1qF`a1CPTVl+|{PSJ*>ne`sGN^pMTft<$WzA*7v~6QuMa?QkaA{+{Aaaws|F*;c;WO z!N#^tvW3@r&QuJZ*1g7+w&IL0duLzKdKF8jh_pnpM)yR+<|0vx`=KU}BD9+D4zxv{ z?01Z+e6CMLh^F#Qmw#ww0aUcI!GB%0-4f+0|6szl@}DyiNX?yjai6%6?E?V!XIm;DIaOo-vQn9RoSsBZ*^f^QUo)DHn&17Lru`VumW-#_F zJt)x4rS)U-HZ#AofT-5j4B*RqdoMY}`>8j}T0H>;#a}LzE^-!X<&V!(8pORH4)_Nd zY^3Adrmy@lAJKh&z-wK8M}LAxSDs)lr^aVe!{RH!`vg_2zd9*^QLo3Db5hS3@a21Y z<%w_HIL-h45bd6C{hg)Dq|5yAb~iJ93gdK_0svs9pVh5qr}t?tRu(ABvwXiDa}yFO z@|u|uRG*zo;b=CUAuYvax14DT{nQbgMeLv_4nE$oE$!UKYch<y*A$g!c;F89|oQ3=*f$`o!o`d7$^$L3ouwRJ_j!%$Om8u zSg4MU@CuF@hd*5JS&b4cIxmbkT>hndCYd>wvS@FjGN!Om!Gx3c&Y?=gehYQGpLi)1 zk2?zpxd|MQRK2DpBq{W~t|x5LNIKx^JLx$+F}@}l`l~x1)xqW$Mmq1rqO;W;jV@jV z>uA?{+IlugyV*ZzY;ldR1`=jUgkKA0?~7pLqTpnI7s<{7i-L_vVMlG#sJKL-1MsS` z*YVIXOo3Fphiu!;CT*L(qT(;QYUF9Td*LU#iBI}K-QuL+7I|5+?Blw8$H`q2d+Q1k z)q0B!CrrZ{|B|3Q`tp*J6FzsHq!7jIDAqA4@DtFB@p(Jpu*<~GbxYBCzL3Wc2&&rp zLv?CP``;c<&`m6M+zmEMN##ubo+1fg$b_E^GB{#2&_ZO`a?<)q2$m`ece5M`Ikfc<19-9SF0eDA-5STf7%3Cgqwq4^ zCt%Y^;*$xOX=qcZ;lSOo7+7vsNp>@z_vyxDn5qutMAcWB1WJbUF~h{o2b0S8iqn1m zY^!rh^DptYE2Hdtob<65-syey#5AhAOa3VTt#Dz(2BrSw3pyOmdRK@h&NX*`ku_Um zwp>fk@}OK@B9wGkZ(Pr8);httvr=3%FIq^gmw`GcWoEU8)SHdb$+)9{`5^hrg zXDeHsC^ezK?vNMPJpCTjQP2h>p(P-GCmi%sC;}0RDmk`0TOi;T7#yT1mV<3dDmi;# zw;-Sy9G>%4$s7Wv!&N{g91RAjz`8;qSBqIUJNJ>_5Ga=#i4nTFwzH#<6R_fBOf;= z6o*}jX^zfRC=U1ija!FbiB8d}+&lqF_*TO3^@ifU=@3Sqh2mEDeWAD&T0ba$h437e z?hVDGb7Dzi5C{gmPgW2_<4{;E3T!6(L$ILfZX|%mpt+)W57Y}P{4GG^QD|=e)j3}O zRXH?|eRYoT3mloN=qDWag}@yT1#))p;v4IwL}yaSThL`gIZ%D z5sTrS8~6ElUr8zoT@2;m%**`+V>FKF|BS-<;eK)Xk&!Yf%t&W#M>6x5c7wdlS#L z>bu648r_eux>lwfZy+DvUm53VVpm#O+**;M+FJ1`8tY;&+L*|(3K$u4iSs1asHi2h z)*qZY4jI-1#g8`*sYhKNOqGvH!*y~@4|uAKIBF3 zR}dsC+xI3)h*>kN)I_}*$;_Ma$;9-P4@DQHb}-X9-!BMS1bcXA8~Wksb160-O6SX00u|WefQ!+#-cR?bC<3``lNPpmWa;CBYrX_| zUKw#{*t)?ffXY{LqD(j(wFUVjwb2<}A|K4e)&5*2W}6~Cctt)r9D6&#dDu1k4MRx% zW{z5$jiYS@QStt#Hm$JRHsOLs1*VlmQ~2Bc4iCz1Ly~No1^npL*VWZx9+`Pdq@EBW z{+3M(ahDnXJJwNK(`qT#owJrjAr_^qrfEDd{;TT2trpiqNx?<;Vy^sZMsnh%>Fn-1 z@B2jQb5xnR{Uw6pj9!nuiF=oJ*(NiZYlkV5HZ{lG#ZuP|&k8=}cBAx4P%_jm{Qw&* zS(cXE{>8QYji1oRQq*b<>>Xt`&9uU!GWbkq@LTFt>71JO9uu#{Chfo1Tpf1$Tkc6k#mX%Y>@n0a#AkB)~*Eh zlA?Z zEUJ4I&K)ZaBaA+H**xN+@jkfSY$YhkwA=ll`}scWlF=Ij1{toqKWwL;8lvgyl#Cx5 z*s((K*2k_M>+g+q8U0P~79a5Nwer5#oS;_T`>eaO`C&lr+aJJt51F3 zpA;oe6Pe#>M*Il3>W)^Wx{Oz; zj#f3a!_MqrkrL%}jhu+MVO&yYrJ6+qXrO@bnP;85Z|Fx^mT42V*txZW`8hJK{LHBG z6St3T=-dnMMF7uw%}K#P#hrqjaY(L>dk8v+`g_maEQoyDApXm zOKW*cMBxe<3)4qOQ24GWDcOe~UABRFxp-HLYy)M6W7uhp?|reBe=c#_Q0lf%;u-(T zlvAMK`8YP_H01G2k8wiP<=YeS)T=8Gs|5SnH;I1J7S1ROlRxxW zs*-^=Gzofa<`THR23_Z_j&|~ul7|MeNK;wXBZHcKSKQh3$FvCx7g z<#yxRN`ooa+lmHKU7Ur7DrM2z)<%R9&osvYat;lzmkrwYwoQGaqX#cvZ)01d&T5Uv z(rlAf9JD*{QER4de&+Crq!hLoY;@CW%_Dxv@Z}Xg2lDr`Y-8n>+2yfv+0b^WM7anR zo`g~25;3c~{>0PoWR;|J40A)f;#8jk7yFVMS>(gryDYi#79IIQjQnVe{P?B-)Y<1K zs&-tDj9LIauv>Gqp{&|e^kntHt!Xl_iEx0Y^*iIz-ca_nurw>QK=Myh<$SRug?e^L zfM}|}0@oj#*Y`)wbvu(61jw?NGuW{6eR`5&T{&(w*V%^a7m>a;usK?5k3?pm7@VK1 z=P!4-JDd60^`C+;4aah4+ecXgGO!{-weu? zGEeLKk2suC-|_6<>NQxmdfvTm$MC8F8K~=^qt#%3k%3F)wLpbm)7(pKqn{nu;h)nH z#HX(hY(CCt-1V$L{HK__zK%eTPrqe5UOP3ev&sB?!g|eHE-#{^jt(Q2so8bWdixRO zA}+8TuNt^noqI#8%5fEY@1t#>b!NvL=RtQL<%a#}t!Gy^ke{gU?;CiM?b1>p{>$pd z0$DMWA;K@8`m~kDuOr&c9P5@t{1q;U>ui4B*KAjOMP9i= zQ_<-bR)TIb@WbOMB2b1w?tgh~JM0fF{?|riuIK&N&+{x2UUoKJy;JF!)AmvXe}oI9 zRp%M+j*IXjtNy-%fXKDa3WVZsN7GHP0d6Z@j4vRSHY|tu^Y4APmO>0^WaxlB`x+ZIxkT-rh#e1WH{&3gJ$$?RRlom~*nUmsA zPki|HG4@DgpMj6wqNa`^sfYdcLiWd=<$d-@I~JF-K__CULu})d(f%TyRNt;)`w+oI zyIx%Yt!Uhr_LR>#2jOE{B&F_cc|&(BXHFFr6if|G)?zcR2KCg&FyGwfieIc~6A$yz zhju;b+|$@Nsnn&b2tD1?qsuOzs2J<&)1q%5p-4*D@}^70Hy?4z;xNRxLSxwm1vF(` z#bv_w{JNfyL9anWM;`C|IyrB2!`87R1^!h&Jx0urWD{N;XICg3(0PILU2Gle4c*{5 zbhPmBc+b?f@7FvgRCWhT+n`xq`~CbmFmBn(#WYfu(K3<65J)0p3=Po;%yRwgHh64w+*Wj}Tj_YG0l z^wvlF`VzKLU$}wla=7B1vvHU7&TZ2sQppw_s;?A#DN>2YRt-$qJ=`60!M)8__jRDc z&_2y#wL2RyW5-lGx58Z07cAAMyuKhf+jqw_x21SZ!Q@9#k5{BLc*k(PkkQ$A#sE z#Ppvr2SWmWp8PN*aOQ-@`?(*&kO;G=kFyI4Ln8i+{4pfb?8%M8LU8hI6zGj1X;Xd% z2Qj1{{Ahe2hBQli7(<#R3}SJ-F=QYsQCAX$0^u}0RS|#=5(xyjK`$6ZfUWfL5Qz#P zqGo;=FAVy}0eK7n(tkcSGuAweGBf>rjQk&YR7BC7JS3nX0|D?i{dJ3}R2XBR29Xz# z5`h_?<`jYi+Kiv`Fd_l*YA!|s=!ma#Fn~a#&le033E+$n0D%7Q06-*?=8FM{WXgOJ zfCNzIy9baEgXa^Is0$q+BfI!L51J7{CJ}$C`ArO&y1-_DLICEw08sv;mO@$JAV8r~ z7Pvs6gUI|d3VtJ|0@MW)Q$hLyXQ Date: Thu, 14 Oct 2021 14:01:12 +0200 Subject: [PATCH 03/23] intro of chapter 4 --- text/problem/main.tex | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/text/problem/main.tex b/text/problem/main.tex index 585f426..c2a1785 100644 --- a/text/problem/main.tex +++ b/text/problem/main.tex @@ -4,17 +4,18 @@ % Crowdsensing applications The plethora of sensors currently embedded in personal devices and other infrastructures have paved the way for the development of numerous \emph{crowdsensing services} (e.g.,~Ring~\cite{ring}, TousAntiCovid~\cite{tousanticovid}, Waze~\cite{waze}, etc.) based on the collected personal, and usually geotagged and timestamped data. % Continuously user-generated data -User--service interactions gather personal event-like data, that are data items comprised of pairs of an identifying attribute of an individual and the---possibly sensitive---information at a timestamp (including contextual information), e.g.,~(\emph{`Bob', `dining', `Canal Saint-Martin', $17{:}00$}). -When the interactions are performed in a continuous manner, we obtain ~\emph{time series} of events. +User--service interactions gather personal event-like data that are data items comprised by pairs of an identifying attribute of an individual and the---possibly sensitive---information at a timestamp (including contextual information), e.g.,~(\emph{`Bob', `dining', `Canal Saint-Martin', $17{:}00$}). +%For a reminder, when the interactions are performed in a continuous manner, we obtain time series of events. % Observation/interaction duration -Depending on the duration, we distinguish the interaction/observation into \emph{finite}, when taking place during a predefined time interval, and \emph{infinite}, when taking place in an uninterrupted fashion. +%Depending on the duration, we distinguish the interaction/observation into finite, when taking place during a predefined time interval, and infinite, when taking place in an uninterrupted fashion. Example~\ref{ex:scenario} shows the result of user--LBS interaction while retrieving location-based information or reporting user-state at various locations. \begin{example} \label{ex:scenario} Consider a finite sequence of spatiotemporal data generated by Bob during an interval of $8$ timestamps, as shown in Figure~\ref{fig:scenario}. - Events in a shade correspond to privacy-sensitive events that Bob has defined beforehand. For instance his home is around {\'E}lys{\'e}e, his workplace is around the Louvre, and his hangout is around Canal Saint-Martin. + Events in a shade correspond to privacy-sensitive + \kat{You should not say that only significant events are privacy sensitive, because then why put noise to the normal timestamps? Maybe say directly significant for the shaded events?} events that Bob has defined beforehand. For instance, $p_1$ and $p_8$ are significant because he was at his home, which is around {\'E}lys{\'e}e, at $p_3$ he was at his workplace around the Louvre, and at $p_5$ he was at his hangout around Canal Saint-Martin. \begin{figure}[htp] \centering @@ -35,10 +36,11 @@ A widely recognized tool that introduces probabilistic randomness to the origina Due to its \emph{composition} property, i.e.,~the combination of differentially private outputs satisfies differential privacy as well, differential privacy is suitable for privacy-preserving time series publishing. \emph{Event}, \emph{user}~\cite{dwork2010differential, dwork2010pan}, and \emph{$w$-event}~\cite{kellaris2014differentially} comprise the possible levels of privacy protection. Event-level limits the privacy protection to \emph{any single event}, user-level protects \emph{all the events} of any user, and $w$-event provides privacy protection to \emph{any sequence of $w$ events}. +\kat{Please write another introduction for your chapter, that is in connection to your thesis, not the paper.. all this information in this paragraph must be said in the introduction of the thesis, not of the chapter.. } -In this chapter, we propose a novel configurable privacy scheme, \emph{\thething} privacy, which takes into account significant events (\emph{\thethings}) in the time series and allocates the available privacy budget accordingly. +In this chapter, we propose a novel configurable privacy scheme, \emph{\thething} privacy (Section~\ref{sec:thething}), which takes into account significant events (\emph{\thethings}) in the time series and allocates the available privacy budget accordingly. We propose three privacy models that guarantee {\thething} privacy. -To further enhance our privacy method, and protect the {\thethings} position in the time series, we propose techniques to perturb the initial {\thethings} set (Section~\ref{sec:theotherthing}). +To further enhance our privacy method, and protect the {\thethings} position in the time series, we propose techniques to perturb the initial {\thethings} set (Section~\ref{sec:theotherthing}).\kat{this is the content that you must enrich and motivate more in the intro of this chapter} \input{problem/thething/main} \input{problem/theotherthing/main} From 7ea2c964b3e09fcd1c00f24e5a0b1e0b0f7d4a82 Mon Sep 17 00:00:00 2001 From: Manos Date: Thu, 14 Oct 2021 14:30:35 +0200 Subject: [PATCH 04/23] evaluation: Final review --- text/evaluation/details.tex | 26 +++++++++++++------------- text/evaluation/main.tex | 2 +- text/evaluation/theotherthing.tex | 16 ++++++++-------- text/evaluation/thething.tex | 22 +++++++++++----------- 4 files changed, 33 insertions(+), 33 deletions(-) diff --git a/text/evaluation/details.tex b/text/evaluation/details.tex index 5ae5a58..fae9afe 100644 --- a/text/evaluation/details.tex +++ b/text/evaluation/details.tex @@ -53,7 +53,7 @@ We sampled the first $1000$ data items of the taxi with identifier `$2$', which We generated synthetic time series of length equal to $100$ timestamps, for which we varied the number and distribution of {\thethings}. In this way, we have a controlled data set that we can use to study the behavior of our proposal. % \kat{more details needed. eg. what is the distributions and number of timestamps used? How many time series you generated? } -We take into account only the temporal order of the points and the position of regular and {\thething} events within the series. +We take into account only the temporal order of the points and the position of regular and {\thething} events within the time series. In Section~\ref{subsec:eval-conf}, we explain in more detail our configuration criteria. % \kat{why is the value not important? at the energy consumption, they mattered} @@ -61,14 +61,14 @@ In Section~\ref{subsec:eval-conf}, we explain in more detail our configuration c \subsection{Configurations} \label{subsec:eval-conf} % \kat{add some info here.. what are the configurations for? What does landmark percentage refer to, and how does it matter? } -We vary the {\thething} percentage (Section~\ref{subsec:eval-conf-lmdk}), i.e.,~the ratio of timestamps that we attribute to {\thethings} and regular events, in order to identify the limitations of our methodology. +We vary the {\thething} percentage (Section~\ref{subsec:eval-conf-lmdk}), i.e.,~the ratio of timestamps that we attribute to {\thethings} and regular events, in order to explore the behavior of our methodology in all possible scenarios. For each data set, we implement a privacy mechanism that injects noise related to the type of its attribute values and we tune the parameters of each mechanism accordingly (Section~\ref{subsec:eval-conf-prv}). -Last but not least, we explain how we generate synthetic data sets with the desired degree of temporal correlation (Section~\ref{subsec:eval-conf-cor}). +Last but not least, we explain how we generate synthetic data sets with various degrees of temporal correlation so as to observe the impact on the overall privacy loss (Section~\ref{subsec:eval-conf-cor}). \subsubsection{{\Thething} percentage} \label{subsec:eval-conf-lmdk} -In the Copenhagen data set, a {\thething} represents a timestamp when a contact device is registered. +In the Copenhagen data set, a {\thething} represents a timestamp when a specific contact device is registered. After identifying the unique contacts within the sample, we achieve each desired {\thethings} to regular events ratio by considering a list that contains a part of these contact devices. In more detail, we achieve $0\%$ {\thethings} by considering an empty list of contact devices, @@ -94,23 +94,23 @@ After analyzing the data and experimenting with different pairs of distance and We generated synthetic data with \emph{skewed} (the {\thethings} are distributed towards the beginning/end of the series), \emph{symmetric} (in the middle), \emph{bimodal} (both end and beginning), and \emph{uniform} (all over the time series) {\thething} distributions. In order to get {\thething} sets with the above distribution features, we generate probability distributions with restricted domain to the beginning and end of the time series, and sample from them, without replacement, the desired number of points. For each case, we place the location, i.e.,~centre, of the distribution accordingly. -That is, for a symmetric we put the location in the middle of the time series and for a left/right skewed to the right/left. -For the bimodal we combine two mirrored skewed distributions. +That is, for symmetric we put the location in the middle of the time series and for left/right skewed to the right/left. +For bimodal we combine two mirrored skewed distributions. Finally, for the uniform distribution we distribute the {\thethings} randomly throughout the time series. -For consistency, we calculate the scale parameter depending on the length of the series by setting it equal to the series' length over a constant. +For consistency, we calculate the scale parameter of the corresponding distribution depending on the length of the time series by setting it equal to the series' length over a constant. \subsubsection{Privacy parameters} \label{subsec:eval-conf-prv} % \kat{Explain why you select each of these perturbation mechanisms for each of the datasets. Is the random response differential private? Mention it! } -For all of te real data sets, we implement $\varepsilon$-differential privacy. +For all of te real data sets, we implement $\varepsilon$-differential privacy by selecting a mechanism, from those that we described in Section~\ref{subsec:prv-mech}, that is best suited for the type of its sensitive attributes. To perturb the contact tracing data of the Copenhagen data set, we utilize the \emph{random response} technique~\cite{wang2017locally}, and at each timestamp we report truthfully, with probability $p = \frac{e^\varepsilon}{e^\varepsilon + 1}$, whether the current contact is a {\thething} or not. -We randomize the energy consumption in HUE with the Laplace mechanism (described in detail in Section~\ref{subsec:prv-mech}). -For T-drive, we perturb the location data wit noise that we sample from the Planar Laplace mechanism~\cite{andres2013geo}. +We randomize the energy consumption in HUE with the Laplace mechanism~\cite{dwork2014algorithmic}. +For T-drive, we perturb the location data with noise that we sample from the Planar Laplace mechanism~\cite{andres2013geo}. We set the privacy budget $\varepsilon = 1$ for all of our experiments and, for simplicity, we assume that for every query sensitivity it holds that $\Delta f = 1$. % \kat{why don't you consider other values as well?} -For the experiments performed on the synthetic data sets, the original values to be released do not influence the outcome of our conclusions, thus we ignore them. +For the experiments that we performed on the synthetic data sets, the original values to be released are not relevant to what we we to observe, and thus we ignore them. % \kat{why are the values not important for the synthetic dataset? This seems a little weird, when said out of context.. our goal is to perturb the values, but do not really care about the way we perturb our values?} % Finally, notice that, depending on the results' variation, most diagrams are in logarithmic scale. @@ -121,12 +121,12 @@ For the experiments performed on the synthetic data sets, the original values to Despite the inherent presence of temporal correlation in time series, it is challenging to correctly discover and quantify it. For this reason, and in order to create a more controlled environment for our experiments, we chose to conduct tests relevant to temporal correlation using synthetic data sets. We model the temporal correlation in the synthetic data as a \emph{stochastic matrix} $P$, using a \emph{Markov Chain}~\cite{gagniuc2017markov}. -$P$ is a $n \times n$ matrix, where the element $P_{ij}$ +$P$ is an $n \times n$ matrix, where the element $P_{ij}$ %at the $i$th row of the $j$th column that represents the transition probability from a state $i$ to another state $j$. %, $\forall i, j \leq n$. It holds that the elements of every row $j$ of $P$ sum up to $1$. -We follow the \emph{Laplacian smoothing} technique~\cite{sorkine2004laplacian} as utilized in~\cite{cao2018quantifying} to generate the matrix $P$ with a degree of temporal correlation $s > 0$ equal to +We follow the \emph{Laplacian smoothing} technique~\cite{sorkine2004laplacian}, as utilized in~\cite{cao2018quantifying}, to generate the matrix $P$ with a degree of temporal correlation $s > 0$ equal to % and generate a stochastic matrix $P$ with a degree of temporal correlation $s$ by calculating each element $P_{ij}$ as follows $$\frac{(I_{n})_{ij} + s}{\sum_{k = 1}^{n}((I_{n})_{jk} + s)}$$ where $I_{n}$ is an \emph{identity matrix} of size $n$. diff --git a/text/evaluation/main.tex b/text/evaluation/main.tex index fae6cd1..7913412 100644 --- a/text/evaluation/main.tex +++ b/text/evaluation/main.tex @@ -3,7 +3,7 @@ In this chapter we present the experiments that we performed in order to evaluate {\thething} privacy (Chapter~\ref{ch:lmdk-prv}) on real and synthetic data sets. Section~\ref{sec:eval-dtl} contains all the details regarding the data sets the we used for our experiments along with the system configurations. Section~\ref{sec:eval-lmdk} evaluates the data utility of the {\thething} privacy mechanisms that we designed in Section~\ref{sec:thething} and investigates the behavior of the privacy loss under temporal correlation for different distributions of {\thethings}. -Section~\ref{sec:eval-lmdk-sel} justifies our decisions while designing the privacy-preserving {\thething} selection component in Section~\ref{sec:theotherthing} and the data utility impact of the latter. +Section~\ref{sec:eval-lmdk-sel} justifies our decisions while designing the privacy-preserving {\thething} selection mechanism in Section~\ref{sec:theotherthing} and the data utility impact of the latter. Finally, Section~\ref{sec:eval-sum} concludes this chapter by summarizing the main results derived from the experiments. \input{evaluation/details} diff --git a/text/evaluation/theotherthing.tex b/text/evaluation/theotherthing.tex index b21c15c..90cb20c 100644 --- a/text/evaluation/theotherthing.tex +++ b/text/evaluation/theotherthing.tex @@ -1,11 +1,11 @@ \section{Selection of landmarks} \label{sec:eval-lmdk-sel} -In this section, we present the experiments on the methodology for the {\thethings} selection presented in Section~\ref{subsec:lmdk-sel-sol}, on the real and synthetic data sets. +In this section, we present the experiments on the methodology for the {\thething} selection presented in Section~\ref{subsec:lmdk-sel-sol}, on the real and synthetic data sets. With the experiments on the synthetic data sets (Section~\ref{subsec:sel-utl}) we show the normalized Euclidean and Wasserstein distance metrics (not to be confused with the temporal distances in Figure~\ref{fig:avg-dist}) % \kat{is this distance the landmark distance that we saw just before ? clarify } of the time series histograms for various distributions and {\thething} percentages. This allows us to justify our design decisions for our concept that we showcased in Section~\ref{subsec:lmdk-sel-sol}. -With the experiments on the real data sets (Section~\ref{subsec:sel-prv}), we show the performance in terms of utility of our three {\thething} mechanisms in combination with the privacy-preserving {\thething} selection mechanism, which enhances the privacy protection of our concept. +With the experiments on the real data sets (Section~\ref{subsec:sel-prv}), we show the performance in terms of utility of our three {\thething} mechanisms in combination with the privacy-preserving {\thething} selection mechanism, which enhances the privacy protection that our concept provides. % \kat{Mention whether it improves the original proposal or not.} @@ -37,7 +37,7 @@ Thus, we choose to utilize the Euclidean distance metric for the implementation \subsection{Privacy budget tuning} \label{subsec:sel-eps} -In Figure~\ref{fig:sel-eps} we test the Uniform mechanism in real data by investing different ratios ($1$\%, $10$\%, $25$\%, and $50$\%) of the available privacy budget $\varepsilon$ in the {\thething} selection mechanism and the remaining to perturbing the data values, in order to figure out the optimal ratio value. +In Figure~\ref{fig:sel-eps} we test the Uniform mechanism in real data by investing different ratios ($1$\%, $10$\%, $25$\%, and $50$\%) of the available privacy budget $\varepsilon$ in the {\thething} selection mechanism and the remaining to perturbing the original data values, in order to figure out the optimal ratio value. Uniform is our baseline implementation, and hence allows us to derive more accurate conclusions in this case. In general, we are expecting to observe that greater ratios will result in more accurate, i.e.,~smaller, {\thething} sets and less accurate values in the released data. @@ -59,8 +59,8 @@ In general, we are expecting to observe that greater ratios will result in more \label{fig:sel-eps} \end{figure} -The application of the randomized response mechanism, in the Copenhagen data set, is tolerant to the fluctuations of the privacy budget and maintains a relatively constant performance in terms of data utility. -For HUE and T-drive, we observe that our implementation performs better for lower ratios, e.g.,~$0.01$, where we end up allocating the majority of the available privacy budget to the data release process instead of the {\thething} selection mechanism. +The application of the randomized response mechanism, in the Copenhagen data set (Figure~\ref{fig:copenhagen-sel-eps}), is tolerant to the fluctuations of the privacy budget and maintains a relatively constant performance in terms of data utility. +For HUE (Figure~\ref{fig:hue-sel-eps}) and T-drive (Figure~\ref{fig:t-drive-sel-eps}), we observe that our implementation performs better for lower ratios, e.g.,~$0.01$, where we end up allocating the majority of the available privacy budget to the data release process instead of the {\thething} selection mechanism. The results of this experiment indicate that we can safely allocate the majority of $\varepsilon$ for publishing the data values, and therefore achieve better data utility, while providing more robust privacy protection to the {\thething} set. @@ -94,7 +94,7 @@ This is natural since we allocated part of the available privacy budget to the p Therefore, there is less privacy budget available for data publishing throughout the time series. % for $0$\% and $100$\% {\thethings}. % \kat{why not for the other percentages?} -Skip performs best in our experiments with HUE, due to the low range in the energy consumption and the high scale of the Laplace noise that it avoids due to the employed approximation. -However, for the Copenhagen data set and T-drive, Skip attains greater mean absolute error than the user-level protection scheme, which exposes no benefit with respect to user-level protection. -Overall, Adaptive has a consistent performance in terms of utility for all of the data sets that we experimented with, and almost always outperforms the user-level privacy. +Skip performs best in our experiments with HUE (Figure~\ref{fig:hue-sel}), due to the low range in the energy consumption and the high scale of the Laplace noise that it avoids due to the employed approximation. +However, for the Copenhagen data set (Figure~\ref{fig:copenhagen-sel}) and T-drive (Figure~\ref{fig:t-drive-sel}), Skip attains high mean absolute error, which exposes no benefit with respect to user-level protection. +Overall, Adaptive has a consistent performance in terms of utility for all of the data sets that we experimented with, and almost always outperforms the user-level privacy protection. Thus, it is selected as the best mechanism to use in general. diff --git a/text/evaluation/thething.tex b/text/evaluation/thething.tex index 2bbb11f..3fd11e7 100644 --- a/text/evaluation/thething.tex +++ b/text/evaluation/thething.tex @@ -10,7 +10,7 @@ We compare with the event- and user-level differential privacy protection levels With the experiments on the synthetic data sets (Section~\ref{subsec:lmdk-expt-cor}) we show the overall privacy loss, % \kat{in the previous set of experiments we were measuring the MAE, now we are measuring the privacy loss... Why is that? Isn't it two sides of the same coin? } -i.e.,~the privacy budget $\varepsilon$, under temporal correlation within our framework when tuning the size and statistical characteristics of the input {\thething} set $L$. +i.e.,~the privacy budget $\varepsilon$ with the extra privacy loss because of the temporal correlation, under temporal correlation within our framework when tuning the size and statistical characteristics of the input {\thething} set $L$. % \kat{mention briefly what you observe} We observe that a greater average {\thething}--regular event distance in a time series can result into greater overall privacy loss under moderate and strong temporal correlation. @@ -52,19 +52,19 @@ overall consistent performance and works best for $60$\% and $80$\% {\thethings} We notice that for $0$\% {\thethings}, it achieves better utility than the event-level protection % \kat{what does this mean? how is it possible?} due to the combination of more available privacy budget per timestamp (due to the absence of {\thethings}) and its adaptive sampling methodology. -The Skip model excels, compared to the others, at cases where it needs to approximate $20$\%, $40$\%, or $100$\% of the times. +Skip excels, compared to the others, at cases where it needs to approximate $20$\%, $40$\%, or $100$\% of the times. % \kat{it seems a little random.. do you have an explanation? (rather few times or all?)} -In general, we notice that, for this data set, it is more beneficial to either invest more privacy budget per event or prefer approximation over introducing randomization. +In general, we notice that, for this data set and due to the application of the random response technique, it is more beneficial to either invest more privacy budget per event or prefer approximation over introducing randomization. -The combination of the small range of measurements in HUE ($[0.28$, $4.45]$ with an average of $0.88$kWh) and the large scale in the Laplace mechanism, allows for schemes that favor approximation over noise injection to achieve a better performance in terms of data utility. -Hence, Skip (Figure~\ref{fig:hue}) achieves a constant low mean absolute error. +The combination of the small range of measurements ($[0.28$, $4.45]$ with an average of $0.88$kWh) in HUE (Figure~\ref{fig:hue}) and the large scale in the Laplace mechanism, allows for mechanisms that favor approximation over noise injection to achieve a better performance in terms of data utility. +Hence, Skip achieves a constant low mean absolute error. % \kat{why?explain} Regardless, the Adaptive mechanism performs by far better than Uniform and % strikes a nice balance\kat{???} balances between event- and user-level protection for all {\thething} percentages. -In the T-drive data set (Figure~\ref{fig:t-drive}), the Adaptive mechanism outperforms Uniform by $10$\%--$20$\% for all {\thething} percentages greater than $40$\% and Skip by more than $20$\%. -The lower density (average distance of $623$m) of the T-drive data set has a negative impact on the performance of Skip; republishing a previous perturbed value is now less accurate than perturbing the new location. +In T-drive (Figure~\ref{fig:t-drive}), the Adaptive mechanism outperforms Uniform by $10$\%--$20$\% for all {\thething} percentages greater than $40$\% and Skip by more than $20$\%. +The lower density (average distance of $623$m) of the T-drive data set has a negative impact on the performance of Skip because republishing a previous perturbed value is now less accurate than perturbing the current location. Principally, we can claim that the Adaptive is the most reliable and best performing mechanism, % with a minimal and generic parameter tuning @@ -83,11 +83,11 @@ result in a more effective budget allocation that would improve the performance \subsection{Temporal distance and correlation} \label{subsec:lmdk-expt-cor} -As previously mentioned, temporal correlation is inherent in continuous publishing, and they are the cause of supplementary privacy loss in the case of privacy-preserving data publication. -In this section, we are interested in studying the effect that the distance of the {\thethings} from every event have on the loss caused by temporal correlation. +As previously mentioned, temporal correlation is inherent in continuous publishing, and it is the cause of supplementary privacy loss in the case of privacy-preserving time series publishing. +In this section, we are interested in studying the effect that the distance of the {\thethings} from every regular event has on the loss caused under the presence of temporal correlation. Figure~\ref{fig:avg-dist} shows a comparison of the average temporal distance of the events from the previous/next {\thething} or the start/end of the time series for various distributions in our synthetic data. -More specifically, we model the distance of an event as the count of the total number of events between itself and the nearest {\thething} or the series edge. +More specifically, we model the distance of an event as the count of the total number of events between itself and the nearest {\thething} or the time series edge. \begin{figure}[htp] \centering @@ -100,7 +100,7 @@ We observe that the uniform and bimodal distributions tend to limit the regular This is due to the fact that the former scatters the {\thethings}, while the latter distributes them on both edges, leaving a shorter space uninterrupted by {\thethings}. % and as a result they reduce the uninterrupted space by landmarks in the sequence. On the contrary, distributing the {\thethings} at one part of the sequence, as in skewed or symmetric, creates a wider space without {\thethings}. -This study provides us with different distance settings that we are going to use in the subsequent temporal leakage study. +This study provides us with different distance settings that we are going to use in the subsequent overall privacy loss study. Figure~\ref{fig:dist-cor} illustrates a comparison among the aforementioned distributions regarding the overall privacy loss under (a)~weak, (b)~moderate, and (c)~strong temporal correlation degrees. The line shows the overall privacy loss---for all cases of {\thething} distribution---without temporal correlation. From 8b0464fdfa5e3ba708ef2c2395c82b676706c679 Mon Sep 17 00:00:00 2001 From: Manos Date: Thu, 14 Oct 2021 14:30:48 +0200 Subject: [PATCH 05/23] text: Minor corrections --- text/preliminaries/correlation.tex | 2 +- text/related/micro.tex | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/text/preliminaries/correlation.tex b/text/preliminaries/correlation.tex index f3c29b9..e71278d 100644 --- a/text/preliminaries/correlation.tex +++ b/text/preliminaries/correlation.tex @@ -230,7 +230,7 @@ The calculation of FPL (Equation~\ref{eq:fpl-2}) becomes: The authors propose solutions to bound the temporal privacy loss, under the presence of weak to moderate correlation, in both finite and infinite data publishing scenarios. In the latter case, they try to find a value for $\varepsilon$ for which the backward and forward privacy loss are equal. -In the former, they similarly try to balance the backward and forward privacy loss while they allocate more $\varepsilon$ at the first and last timestamps, since they have higher impact to the privacy loss of the next and previous ones. +In the former, they similarly try to balance the backward and forward privacy loss while they allocate more $\varepsilon$ at the first and last timestamps, since they have higher impact on the privacy loss of the next and previous ones. This way they achieve an overall constant temporal privacy loss throughout the time series. According to the technique's intuition, stronger correlation result in higher privacy loss. diff --git a/text/related/micro.tex b/text/related/micro.tex index b654c02..b411576 100644 --- a/text/related/micro.tex +++ b/text/related/micro.tex @@ -433,7 +433,7 @@ This calculation is done for each individual that is included in the original da The backward/forward privacy loss at any time point depends on the backward/forward privacy loss at the previous/next instance, the backward/forward temporal correlations, and $\varepsilon$. The authors propose solutions to bound the temporal privacy loss, under the presence of weak to moderate correlations, in both finite and infinite data publishing scenarios. In the latter case, they try to find a value for $\varepsilon$ for which the backward and forward privacy loss are equal. -In the former, they similarly try to balance the backward and forward privacy loss while they allocate more $\varepsilon$ at the first and last time points, since they have higher impact to the privacy loss of the next and previous ones. +In the former, they similarly try to balance the backward and forward privacy loss while they allocate more $\varepsilon$ at the first and last time points, since they have higher impact on the privacy loss of the next and previous ones. This way they achieve an overall constant temporal privacy loss throughout the time series. According to the technique's intuition, stronger correlations result in higher privacy loss. However, the loss is smaller when the dimension of the transition matrix, which is extracted according to the modeling of the correlations (here it is Markov chain), is larger due to the fact that larger transition matrices tend to be uniform, resulting in weaker data dependence. From bb8346f2d6714c85b5a545bb1d3011a6155ccfd9 Mon Sep 17 00:00:00 2001 From: katerinatzo Date: Thu, 14 Oct 2021 16:15:32 +0200 Subject: [PATCH 06/23] working on 4.1 --- text/problem/thething/main.tex | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/text/problem/thething/main.tex b/text/problem/thething/main.tex index 4584ad4..fe60e98 100644 --- a/text/problem/thething/main.tex +++ b/text/problem/thething/main.tex @@ -1,17 +1,19 @@ \section{Significant events} \label{sec:thething} -The privacy mechanisms for the aforementioned levels assume that in a time series any single event, or any sequence of events, or the entire series of events is equally privacy-significant for the users. -In reality, this is an simplistic assumption. -The significance of an event is related to certain user-defined privacy criteria, or to its adjacent events, as well as to the entire time series. -We term significant events as \emph{{\thething} events} or simply \emph{\thethings}. -Identifying {\thethings} can be done in an automatic or manual way (but is out of scope for this work). +The privacy mechanisms for the user, w-event and event levels that are already proposed in the literature, assume that in a time series any single event, or any sequence of events, or the entire series of events is equally privacy-significant for the users. +In reality, this is a simplistic\kat{I would not say simplistic, but unrealistic assumption that deteriorates unnecessarily the quality of the perturbed data} assumption. +The fact that an event is significant, can be related to certain user-defined privacy criteria, or to its adjacent events, as well as to the entire time series. +We term significant events as \emph{{\thething} events} or simply \emph{\thethings}, following relevant literature\kat{can you find some other work that uses the same term? otherwise one can raise the question why not ot use the word significant }. + +Identifying {\thethings} in timeseries can be done in an automatic or manual way. For example, in spatiotemporal data, \emph{places where an individual spent some time} denote \emph{points of interest} (POIs) (called also stay points)~\cite{zheng2015trajectory}. Such events, and more particularly their spatial attribute values, can be less privacy-sensitive~\cite{primault2018long}, e.g.,~parks, theaters, etc. or, if individuals frequent them, they can reveal supplementary information, e.g.,~residences (home addresses)~\cite{gambs2010show}, places of worship (religious beliefs)~\cite{franceschi-bicchierairussell2015redditor}, etc. POIs can be an example of how we can choose {\thethings}, but the idea is not limited to these. Another example is the detection of privacy-sensitive user interactions by \emph{contact tracing} applications. This can be practical in decease control~\cite{eames2003contact}, similar to the recent outbreak of the Coronavirus disease 2019 (COVID-19) epidemic~\cite{ahmed2020survey}. Last but not least, {\thethings} in \emph{smart grid} electricity usage patterns could not only reveal the energy consumption of a user but also information regarding activities, e.g.,~`at work', `sleeping', etc. and types of appliances already installed or recently purchased~\cite{khurana2010smart}. +We stress out that {\thething} identification is an orthogonal problem to ours, and that we consider {\thethings} given as input to our problem. \begin{example} \label{ex:st-cont} From 270483de67aecfe1fe8f697ad7f7cb5ebed84774 Mon Sep 17 00:00:00 2001 From: Manos Date: Thu, 14 Oct 2021 17:17:31 +0200 Subject: [PATCH 07/23] evaluation: Minor corrections --- text/evaluation/details.tex | 5 ++--- text/evaluation/theotherthing.tex | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/text/evaluation/details.tex b/text/evaluation/details.tex index fae9afe..e27fea9 100644 --- a/text/evaluation/details.tex +++ b/text/evaluation/details.tex @@ -1,4 +1,4 @@ -\section{Experimental setting and data sets} +\section{Setting, configurations, and data sets} \label{sec:eval-dtl} In this section we list all the relevant details regarding the evaluation setting (Section~\ref{subsec:eval-setup}), and we present the real and synthetic data sets that we used (Section~\ref{subsec:eval-dat}), along with the corresponding configurations (Section~\ref{subsec:eval-conf}). @@ -123,8 +123,7 @@ For this reason, and in order to create a more controlled environment for our ex We model the temporal correlation in the synthetic data as a \emph{stochastic matrix} $P$, using a \emph{Markov Chain}~\cite{gagniuc2017markov}. $P$ is an $n \times n$ matrix, where the element $P_{ij}$ %at the $i$th row of the $j$th column that -represents the transition probability from a state $i$ to another state $j$. -%, $\forall i, j \leq n$. +represents the transition probability from a state $i$ to another state $j$, $\forall$ $i$, $j$ $\leq$ $n$. It holds that the elements of every row $j$ of $P$ sum up to $1$. We follow the \emph{Laplacian smoothing} technique~\cite{sorkine2004laplacian}, as utilized in~\cite{cao2018quantifying}, to generate the matrix $P$ with a degree of temporal correlation $s > 0$ equal to % and generate a stochastic matrix $P$ with a degree of temporal correlation $s$ by calculating each element $P_{ij}$ as follows diff --git a/text/evaluation/theotherthing.tex b/text/evaluation/theotherthing.tex index 90cb20c..4ba8f9c 100644 --- a/text/evaluation/theotherthing.tex +++ b/text/evaluation/theotherthing.tex @@ -1,4 +1,4 @@ -\section{Selection of landmarks} +\section{Selection of {\thethings}} \label{sec:eval-lmdk-sel} In this section, we present the experiments on the methodology for the {\thething} selection presented in Section~\ref{subsec:lmdk-sel-sol}, on the real and synthetic data sets. With the experiments on the synthetic data sets (Section~\ref{subsec:sel-utl}) we show the normalized Euclidean and Wasserstein distance metrics (not to be confused with the temporal distances in Figure~\ref{fig:avg-dist}) From 5dda0700fb43e3545c48904f0b3652cf1164b093 Mon Sep 17 00:00:00 2001 From: Manos Date: Thu, 14 Oct 2021 18:18:23 +0200 Subject: [PATCH 08/23] text: Minor corrections --- text/problem/summary.tex | 2 +- text/problem/thething/main.tex | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/text/problem/summary.tex b/text/problem/summary.tex index a44f282..5c83625 100644 --- a/text/problem/summary.tex +++ b/text/problem/summary.tex @@ -3,7 +3,7 @@ In this chapter, we presented \emph{{\thething} privacy} for privacy-preserving time series publishing, which allows for the protection of significant events, while improving the utility of the final result with respect to the traditional user-level differential privacy. We also proposed three models for {\thething} privacy, and quantified the privacy loss under temporal correlation. -Furthermore, we present three solutions to enhance our privacy scheme by protecting the actual temporal position of the{\thethings} in the time series. +Furthermore, we present three solutions to enhance our privacy scheme by protecting the actual temporal position of the {\thethings} in the time series. We differ the experimental evaluation of our methodology to Chapter~\ref{ch:eval} we experiment with real and synthetic data sets to demonstrate the applicability of the {\thething} privacy models by themselves (Section~\ref{sec:eval-lmdk-sel}) and in combination with the {\thething} selection component (Section~\ref{sec:eval-lmdk}). %Our experiments on real and synthetic data sets validate our proposal. diff --git a/text/problem/thething/main.tex b/text/problem/thething/main.tex index fe60e98..1573a54 100644 --- a/text/problem/thething/main.tex +++ b/text/problem/thething/main.tex @@ -6,7 +6,7 @@ In reality, this is a simplistic\kat{I would not say simplistic, but unrealistic The fact that an event is significant, can be related to certain user-defined privacy criteria, or to its adjacent events, as well as to the entire time series. We term significant events as \emph{{\thething} events} or simply \emph{\thethings}, following relevant literature\kat{can you find some other work that uses the same term? otherwise one can raise the question why not ot use the word significant }. -Identifying {\thethings} in timeseries can be done in an automatic or manual way. +Identifying {\thethings} in time series can be done in an automatic or manual way. For example, in spatiotemporal data, \emph{places where an individual spent some time} denote \emph{points of interest} (POIs) (called also stay points)~\cite{zheng2015trajectory}. Such events, and more particularly their spatial attribute values, can be less privacy-sensitive~\cite{primault2018long}, e.g.,~parks, theaters, etc. or, if individuals frequent them, they can reveal supplementary information, e.g.,~residences (home addresses)~\cite{gambs2010show}, places of worship (religious beliefs)~\cite{franceschi-bicchierairussell2015redditor}, etc. POIs can be an example of how we can choose {\thethings}, but the idea is not limited to these. From fd8e23b3a8df10b48ddd39d0e4aed94e485d3464 Mon Sep 17 00:00:00 2001 From: katerinatzo Date: Thu, 14 Oct 2021 19:31:52 +0200 Subject: [PATCH 09/23] working on 4.1 --- text/problem/thething/main.tex | 59 ++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 28 deletions(-) diff --git a/text/problem/thething/main.tex b/text/problem/thething/main.tex index 1573a54..e768fd0 100644 --- a/text/problem/thething/main.tex +++ b/text/problem/thething/main.tex @@ -8,45 +8,48 @@ We term significant events as \emph{{\thething} events} or simply \emph{\thethin Identifying {\thethings} in time series can be done in an automatic or manual way. For example, in spatiotemporal data, \emph{places where an individual spent some time} denote \emph{points of interest} (POIs) (called also stay points)~\cite{zheng2015trajectory}. -Such events, and more particularly their spatial attribute values, can be less privacy-sensitive~\cite{primault2018long}, e.g.,~parks, theaters, etc. or, if individuals frequent them, they can reveal supplementary information, e.g.,~residences (home addresses)~\cite{gambs2010show}, places of worship (religious beliefs)~\cite{franceschi-bicchierairussell2015redditor}, etc. +Such events, and more particularly their spatial attribute values, can be less privacy-sensitive~\cite{primault2018long}, e.g.,~parks, theaters, etc., or, if individuals frequent them, they can reveal supplementary information, e.g.,~residences (home addresses)~\cite{gambs2010show}, places of worship (religious beliefs)~\cite{franceschi-bicchierairussell2015redditor}, etc. POIs can be an example of how we can choose {\thethings}, but the idea is not limited to these. Another example is the detection of privacy-sensitive user interactions by \emph{contact tracing} applications. This can be practical in decease control~\cite{eames2003contact}, similar to the recent outbreak of the Coronavirus disease 2019 (COVID-19) epidemic~\cite{ahmed2020survey}. -Last but not least, {\thethings} in \emph{smart grid} electricity usage patterns could not only reveal the energy consumption of a user but also information regarding activities, e.g.,~`at work', `sleeping', etc. and types of appliances already installed or recently purchased~\cite{khurana2010smart}. +Last but not least, {\thethings} in \emph{smart grid} electricity usage patterns may not only reveal the energy consumption of a user but also information regarding activities, e.g.,~`at work', `sleeping', etc., or types of appliances already installed or recently purchased~\cite{khurana2010smart}. We stress out that {\thething} identification is an orthogonal problem to ours, and that we consider {\thethings} given as input to our problem. -\begin{example} - \label{ex:st-cont} - - Figure~\ref{fig:st-cont} shows the case when we want to protect all of Bob's significant events ($p_1$, $p_3$, $p_5$, $p_8$) in his trajectory shown in Figure~\ref{fig:scenario}. - % That is, we have to allocate privacy budget $\varepsilon$ such that at any timestamp $t$ it holds that $\varepsilon_t + \varepsilon_1 + \varepsilon_3 + \varepsilon_5 + \varepsilon_8 \leq \varepsilon$. - In this scenario, event-level protection is not suitable since it can only protect one event at a time. - Hence, we have to apply user-level privacy protection by distributing equal portions of $\varepsilon$ to all the events, i.e.,~$\frac{\varepsilon}{8}$ to each one (the equivalent of applying $8$-event privacy). - In this way, we have protected the {\thething} points; we have allocated a total of $\frac{\varepsilon}{2}<\varepsilon$ to the {\thethings}. - - \begin{figure}[htp] - \centering - \includegraphics[width=\linewidth]{problem/st-cont} - \caption{User-level and {\thething} $\varepsilon$-differential privacy protection for the time series of Figure~\ref{fig:scenario}.} - \label{fig:st-cont} - \end{figure} - - However, perturbing by $\frac{\varepsilon}{8}$ each regular point deteriorates the data utility unnecessarily. - Notice that the overall privacy budget that we ended up allocating to the user-defined significant events is equal to $\frac{\varepsilon}{2}$ and leaves an equal amount of budget to distribute to any current event. - In other words, uniformly allocating $\frac{\varepsilon}{5}$ to every event would still achieve the Bob's privacy goal, i.e.,~protect every significant event, while achieving better utility overall. - -\end{example} - -We argue that protecting only {\thething} events along with any regular event release is sufficient for the user's protection, while it improves data utility. -Considering {\thething} events can prevent over-perturbing the data in the benefit of their final quality. +We argue that protecting only {\thething} events along with any regular event release -- instead of protecting every event in the timeseries -- is sufficient for the user's protection, while it improves data utility. +More specifically, important events are adequately protected, while less important ones are not excessively perturbed. \kat{something feels wrong with this statement, because in figure 2 regular and landmarks seem to receive the same amount of noise..} +%In fact, considering {\thething} events can prevent over-perturbing the data in the benefit of their final quality. Take for example the scenario in Figure~\ref{fig:st-cont}, where {\thethings} are highlighted in gray. If we want to protect the {\thething} points, we have to allocate at most a budget of $\varepsilon$ to the {\thethings}, while saving some for the release of regular events. Essentially, the more budget we allocate to an event the less we protect it, but at the same time we maintain its utility. With {\thething} privacy we propose to distribute the budget taking into account only the existence of the {\thethings} when we release an event of the time series, i.e.,~allocating $\frac{\varepsilon}{5}$ ($4\ \text{\thethings} + 1\ \text{regular point}$) to each event (see Figure~\ref{fig:st-cont}). -This way, we still guarantee that the {\thethings} are adequately protected, as they receive a total budget of $\frac{4\varepsilon}{5}<\varepsilon$. -At the same time, we avoid over-perturbing the regular events, as we allocate to them a higher total budget ($\frac{4\varepsilon}{5}$) than in user-level ($\frac{\varepsilon}{2}$), and thus less noise. +This way, we still guarantee\footnote{$\epsilon$-differential privacy guarantees that the allocated budget should be less or equal to $\epsilon$, and not precisely how much.\kat{Mano check.}} that the {\thethings} are adequately protected, as they receive a total budget of $\frac{4\varepsilon}{5}<\varepsilon$. +At the same time, we avoid over-perturbing the regular events, as we allocate to them a higher total budget ($\frac{4\varepsilon}{5}$) compared to the user-level scenario ($\frac{\varepsilon}{2}$), and thus less noise. + + +\begin{example} + \label{ex:st-cont} + + Figure~\ref{fig:st-cont} shows the case when we want to protect all of Bob's significant events ($p_1$, $p_3$, $p_5$, $p_8$) in his trajectory shown in Figure~\ref{fig:scenario}. + % That is, we have to allocate privacy budget $\varepsilon$ such that at any timestamp $t$ it holds that $\varepsilon_t + \varepsilon_1 + \varepsilon_3 + \varepsilon_5 + \varepsilon_8 \leq \varepsilon$. + In this scenario, event-level protection is not suitable since it can only protect one event at a time. + Hence, we have to apply user-level privacy protection by distributing equal portions of $\varepsilon$ to all the events, i.e.,~$\frac{\varepsilon}{8}$ to each one (the equivalent of applying $8$-event privacy). + In this way, we have protected the {\thething} points; we have allocated a total of $\frac{\varepsilon}{2}<\varepsilon$ to the {\thethings}. + + \begin{figure}[htp] + \centering + \includegraphics[width=\linewidth]{problem/st-cont} + \caption{User-level and {\thething} $\varepsilon$-differential privacy protection for the time series of Figure~\ref{fig:scenario}.} + \label{fig:st-cont} + \end{figure} + + However, perturbing by $\frac{\varepsilon}{8}$ each regular point deteriorates the data utility unnecessarily. + Notice that the overall privacy budget that we ended up allocating to the user-defined significant events is equal to $\frac{\varepsilon}{2}$ and leaves an equal amount of budget to distribute to any current event. + In other words, uniformly allocating $\frac{\varepsilon}{5}$ to every event would still achieve the Bob's privacy goal, i.e.,~protect every significant event, while achieving better utility overall. + +\end{example} + \input{problem/thething/contribution} \input{problem/thething/problem} \input{problem/thething/solution} From 5b011b1d6a26cb56eea64e789b5cc6e8732a103a Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 00:00:14 +0200 Subject: [PATCH 10/23] abstract: Review --- text/abstract.tex | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/text/abstract.tex b/text/abstract.tex index 68b39bf..507bff8 100644 --- a/text/abstract.tex +++ b/text/abstract.tex @@ -1,27 +1,31 @@ \chapter{Abstract} \label{ch:abs} -\kat{Il faut aussi en francais :) } +% \kat{Il faut aussi en francais :) } +% \mk{D'accord :( } Sensors, portable devices, and location-based services, generate massive amounts of geo-tagged, and/or location- and user-related data on a daily basis. The manipulation of such data is useful in numerous application domains, e.g.,~healthcare, intelligent buildings, and traffic monitoring. -A high percentage of these data carry information of users' activities and other personal details, and thus their manipulation and sharing arise concerns about the privacy of the individuals involved. -To enable the secure---from the users' privacy perspective---data sharing, researchers have already proposed various seminal techniques for the protection of users' privacy. +A high percentage of these data carry information of user activities and other personal details, and thus their manipulation and sharing arise concerns about the privacy of the individuals involved. +To enable the secure---from the user privacy perspective---data sharing, researchers have already proposed various seminal techniques for the protection of user privacy. However, the continuous fashion in which data are generated nowadays, and the high availability of external sources of information, pose more threats and add extra challenges to the problem. -\kat{Mention here the extra challenges posed by the specific problem that you address : the Landmark privacy} +% \kat{Mention here the extra challenges posed by the specific problem that you address : the Landmark privacy} +It is therefore essential to design solutions that not only guarantee privacy protection but also provide configurability and account the preferences of the users. % Survey -In this thesis, we visit the works done on data privacy for continuous data publishing, and report on the proposed solutions, with a special focus on solutions concerning location or geo-referenced data. -As a matter of fact, a wealth of algorithms have been proposed for privacy-preserving data publishing, either for microdata or statistical data. +In this thesis, we investigate the literature regarding data privacy in continuous data publishing, and report on the proposed solutions, with a special focus on solutions concerning location or geo-referenced data. +As a matter of fact, a wealth of algorithms has been proposed for privacy-preserving data publishing, either for microdata or statistical data. In this context, we seek to offer a guide that would allow readers to choose the proper algorithm(s) for their specific use case accordingly. -We provide an insight into time-related properties of the algorithms, e.g.,~if they work on infinite, real-time data, or if they take into consideration existing data dependencies. - +We provide an insight into time-related properties of the algorithms, e.g.,~if they work on infinite, real-time data, or if they take into consideration existing data dependence. % Landmarks -Having discussed the literature around continuous data publication, we continue to propose a novel type of data privacy, called \emph{\thething} privacy. +Having discussed the literature around continuous data publishing, we continue to propose a novel type of data privacy, called \emph{{\thething} privacy}. We argue that in continuous data publishing, events are not equally significant in terms of privacy, and hence they should affect the privacy-preserving processing differently. Differential privacy is a well-established paradigm in privacy-preserving time series publishing. Different schemes exist, protecting either a single timestamp, or all the data per user or per window in the time series, considering however all timestamps as equally significant. -The novel scheme that we propose, \emph{\thething} privacy,is based on differential privacy, but also takes into account significant events (\emph{\thethings}) in the time series and allocates the available privacy budget accordingly. -We design three privacy models that guarantee {\thething} privacy and validate our proposal on real and synthetic data sets. \kat{add selection, and a small comment on the conclusions driven by the experiments.} +The novel scheme that we propose, {\thething} privacy, is based on differential privacy, but also takes into account significant events (\emph{\thethings}) in the time series and allocates the available privacy budget accordingly. +We design three privacy schemes that guarantee {\thething} privacy and further extend them in order to provide more robust privacy protection to the {\thething} set. +We evaluate our proposal on real and synthetic data sets and assess the impact on data utility with emphasis on situations under the presence of temporal correlation. +% \kat{add selection, and a small comment on the conclusions driven by the experiments.} +The results of the experimental evaluation and comparative analysis of {\thething} privacy validate its applicability to several use case scenarios with and without the presence of temporal correlation. \paragraph{Keywords:} From 033c4afcd4b8011a330a3465274e91245e73f145 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 00:16:25 +0200 Subject: [PATCH 11/23] abstract: Omelette du fromage --- text/resume.tex | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 text/resume.tex diff --git a/text/resume.tex b/text/resume.tex new file mode 100644 index 0000000..0bd353f --- /dev/null +++ b/text/resume.tex @@ -0,0 +1,25 @@ +\chapter{Résumé} +\label{ch:res} +Les capteurs, les appareils portables et les services basés sur la localisation génèrent quotidiennement des quantités massives de données géolocalisées et/ou liées à la localisation et aux utilisateurs. +La manipulation de ces données est utile dans de nombreux domaines d'application, e.g.,~les soins de santé, les bâtiments intelligents, et la surveillance du trafic. +Un pourcentage élevé de ces données contient des informations sur les activités des utilisateurs et d'autres détails personnels, et donc leur manipulation et leur partage soulèvent des inquiétudes quant à la confidentialité des personnes concernées. +Cependant, la manière continue avec laquelle les données sont générées de nos jours et la haute disponibilité de sources d'information externes posent davantage de menaces et ajoutent des défis supplémentaires au problème. +Il est donc essentiel de concevoir des solutions qui non seulement garantissent la protection de la confidentialité, mais offrent également une configurabilité et tiennent compte des préférences des utilisateurs. + +Dans cette thèse, nous étudions la littérature concernant la confidentialité des données dans la publication de données en continu, et rapportons les solutions proposées, avec un accent particulier sur les solutions concernant la localisation ou les données géo-référencées. +En fait, une multitude d'algorithmes ont été proposés pour la publication de données préservant la confidentialité, que ce soit pour des microdonnées ou des données statistiques. +Dans ce contexte, nous cherchons à offrir un guide qui permettrait aux lecteurs de choisir en conséquence le ou les algorithmes appropriés pour leur cas d'utilisation spécifique. +Nous donnons un aperçu des propriétés temporelles des algorithmes, e.g.,~s'ils fonctionnent sur des données infinies en temps réel, ou s'ils prennent en considération la dépendance des données existantes. + +Après avoir discuté de la littérature sur la publication continue des données, nous continuons à proposer un nouveau type de confidentialité des données, appelé \emph{confidentialité {\thething}}. +Nous soutenons que dans la publication continue de données, les événements ne sont pas aussi importants en termes de confidentialité et, par conséquent, ils devraient affecter différemment le traitement préservant la confidentialité. +La confidentialité différentielle est un paradigme bien établi dans la publication de séries chronologiques préservant la confidentialité. +Différents schémas existent, protégeant soit un seul horodatage, soit toutes les données par utilisateur ou par fenêtre dans la série temporelle, considérant cependant tous les horodatages comme également significatifs. +Le nouveau schéma que nous proposons, confidentialité {\thething}, est basé sur une confidentialité différentielle, mais prend également en compte les événements significatifs (\emph{\thethings}) dans la série chronologique et alloue le budget de confidentialité disponible en conséquence. +Nous concevons trois schémas de confidentialité qui garantissent la confidentialité de {\thething} et les étendons davantage afin de fournir une protection de confidentialité plus robuste à l'ensemble {\thething}. +Nous évaluons notre proposition sur des ensembles de données réelles et synthétiques et évaluons l'impact sur l'utilité des données en mettant l'accent sur les situations en présence de corrélation temporelle. +Les résultats de l'évaluation expérimentale et de l'analyse comparative de la confidentialité de {\thething} valident son applicabilité à plusieurs scénarios de cas d'utilisation avec et sans la présence de corrélation temporelle. + + +\paragraph{Mots clés :} +confidentialité des informations, publication continue des données, crowdsensing, traitement des données préservant la confidentialité From 1d6025e9da3b59a605301ce21f3cb71a0793e329 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 00:16:38 +0200 Subject: [PATCH 12/23] abstract: Minor correction --- text/abstract.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/abstract.tex b/text/abstract.tex index 507bff8..230b98d 100644 --- a/text/abstract.tex +++ b/text/abstract.tex @@ -4,7 +4,7 @@ % \mk{D'accord :( } Sensors, portable devices, and location-based services, generate massive amounts of geo-tagged, and/or location- and user-related data on a daily basis. The manipulation of such data is useful in numerous application domains, e.g.,~healthcare, intelligent buildings, and traffic monitoring. -A high percentage of these data carry information of user activities and other personal details, and thus their manipulation and sharing arise concerns about the privacy of the individuals involved. +A high percentage of these data carry information of user activities and other personal details, and thus their manipulation and sharing raise concerns about the privacy of the individuals involved. To enable the secure---from the user privacy perspective---data sharing, researchers have already proposed various seminal techniques for the protection of user privacy. However, the continuous fashion in which data are generated nowadays, and the high availability of external sources of information, pose more threats and add extra challenges to the problem. % \kat{Mention here the extra challenges posed by the specific problem that you address : the Landmark privacy} From fb60f3ef74b204115ace5a6e2130f3cf4f665f64 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 00:19:48 +0200 Subject: [PATCH 13/23] abstract: Correction mineure --- text/resume.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/resume.tex b/text/resume.tex index 0bd353f..6330cf3 100644 --- a/text/resume.tex +++ b/text/resume.tex @@ -16,9 +16,9 @@ Nous soutenons que dans la publication continue de données, les événements ne La confidentialité différentielle est un paradigme bien établi dans la publication de séries chronologiques préservant la confidentialité. Différents schémas existent, protégeant soit un seul horodatage, soit toutes les données par utilisateur ou par fenêtre dans la série temporelle, considérant cependant tous les horodatages comme également significatifs. Le nouveau schéma que nous proposons, confidentialité {\thething}, est basé sur une confidentialité différentielle, mais prend également en compte les événements significatifs (\emph{\thethings}) dans la série chronologique et alloue le budget de confidentialité disponible en conséquence. -Nous concevons trois schémas de confidentialité qui garantissent la confidentialité de {\thething} et les étendons davantage afin de fournir une protection de confidentialité plus robuste à l'ensemble {\thething}. +Nous concevons trois schémas de confidentialité qui garantissent la confidentialité {\thething} et les étendons davantage afin de fournir une protection de confidentialité plus robuste à l'ensemble {\thething}. Nous évaluons notre proposition sur des ensembles de données réelles et synthétiques et évaluons l'impact sur l'utilité des données en mettant l'accent sur les situations en présence de corrélation temporelle. -Les résultats de l'évaluation expérimentale et de l'analyse comparative de la confidentialité de {\thething} valident son applicabilité à plusieurs scénarios de cas d'utilisation avec et sans la présence de corrélation temporelle. +Les résultats de l'évaluation expérimentale et de l'analyse comparative de la confidentialité {\thething} valident son applicabilité à plusieurs scénarios de cas d'utilisation avec et sans la présence de corrélation temporelle. \paragraph{Mots clés :} From acc745cde19b0f335669943234eb93c55132ec80 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 01:51:55 +0200 Subject: [PATCH 14/23] acknowledgements: *Plays little violin* --- text/acknowledgements.tex | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/text/acknowledgements.tex b/text/acknowledgements.tex index d1f2dd5..8b80fc4 100644 --- a/text/acknowledgements.tex +++ b/text/acknowledgements.tex @@ -1,14 +1,16 @@ \chapter{Acknowledgements} \label{ch:ack} -\mk{WIP} -Upon the completion of my thesis, I would like to express my deep gratitude to my research supervisors for their patient guidance, enthusiastic encouragement and useful critiques of this research work. -I would also like to thank the reporters for their feedback, comments, and time. +Upon the completion of my thesis, I would like to express my deep gratitude to Prof. Dimitris Kotzinos for believing in me and for providing me with opportunities that helped me pave my path in academia. -% \kat{the jury and the reporters do not contribute; thank them for their feedback, comments and time} +This thesis would not have been possible without the patient guidance of Prof. Katerina Tzompanaki. +Her love for learning and hard work were inspiring and served as the catalyst for every single step that I made towards getting a grasp on computer science. -A special thanks to my department’s faculty, staff and fellow researchers for their valuable assistance whenever needed and for creating a pleasant and creative environment during my studies. +% I am really grateful to the reporters for their time and valuable feedback. -Last but not least, I wish to thank my family and friends for their unconditional support and encouragement all these years. +A special thanks goes to Alexandros Kontarinis for being an exemplary colleague and a unique friend. +I would also like to thank the department's faculty, the lab's staff and researchers for creating a pleasant and creative environment. + +Last but not least, I wish to express my thankfulness to my family and friends for their unconditional support and encouragement all these years. \bigskip \noindent From b9ab8c8cae10d2d986233bf954a0455c014f7ccc Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 02:04:23 +0200 Subject: [PATCH 15/23] acknowledgements: *More violin* --- text/acknowledgements.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/acknowledgements.tex b/text/acknowledgements.tex index 8b80fc4..46aeae1 100644 --- a/text/acknowledgements.tex +++ b/text/acknowledgements.tex @@ -5,9 +5,9 @@ Upon the completion of my thesis, I would like to express my deep gratitude to P This thesis would not have been possible without the patient guidance of Prof. Katerina Tzompanaki. Her love for learning and hard work were inspiring and served as the catalyst for every single step that I made towards getting a grasp on computer science. -% I am really grateful to the reporters for their time and valuable feedback. +I am genuinely grateful to the reporters for their time, effort, and valuable feedback. -A special thanks goes to Alexandros Kontarinis for being an exemplary colleague and a unique friend. +A special thanks goes to Alexandros Kontarinis for being an exemplary colleague and a unique companion during this journey. I would also like to thank the department's faculty, the lab's staff and researchers for creating a pleasant and creative environment. Last but not least, I wish to express my thankfulness to my family and friends for their unconditional support and encouragement all these years. From f44a2d590aa163db504a070fb6e3fc8a1d6e2fc6 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 08:58:54 +0200 Subject: [PATCH 16/23] bibliography: Update --- text/bibliography.bib | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/text/bibliography.bib b/text/bibliography.bib index 0eb4fca..e190cfe 100644 --- a/text/bibliography.bib +++ b/text/bibliography.bib @@ -17,7 +17,7 @@ @online{acxiom, title = {Acxiom}, url = {https://acxiom.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @book{adler2010geometry, @@ -268,7 +268,7 @@ year = {2016}, publisher = {Chambers and Partners}, howpublished = {\url{https://chambersandpartners.com/article/713/personal-data-the-new-oil-of-the-digital-economy}}, - note = {Accessed on December 1, 2019} + note = {Accessed on October 11, 2021} } @article{chan2011private, @@ -299,7 +299,7 @@ publisher = {Channel 4}, year = {2018}, howpublished = {\url{https://channel4.com/news/data-democracy-and-dirty-tricks-cambridge-analytica-uncovered-investigation-expose}}, - note = {Accessed on December 1, 2019} + note = {Accessed on October 11, 2021} } @inproceedings{chatzikokolakis2015geo, @@ -548,7 +548,7 @@ publisher = {The Economist}, year = {2016}, howpublished = {\url{https://economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data}}, - note = {Accessed on December 1, 2019} + note = {Accessed on October 11, 2021} } @inproceedings{efthymiou2015big, @@ -598,13 +598,13 @@ @online{experian, title = {Experian}, url = {https://experian.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @online{facebook, title = {Facebook}, \url = {https://facebook.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @inproceedings{fan2013differentially, @@ -646,7 +646,7 @@ @online{foursquare, title = {Foursquare}, url = {https://foursquare.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @misc{franceschi-bicchierairussell2015redditor, @@ -655,7 +655,7 @@ year = {2015}, publisher = {Mashable}, howpublished = {\url{https://mashable.com/2015/01/28/redditor-muslim-cab-drivers}}, - note = {Accessed on July 1, 2020} + note = {Accessed on October 11, 2021} } @book{fuller2009introduction, @@ -751,7 +751,7 @@ @online{gmaps, title = {Google Maps}, url = {https://google.com/maps}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @article{goldreich1998secure, @@ -1345,7 +1345,7 @@ @online{osm, title = {OpenStreetMap}, url = {https://openstreetmap.org}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @article{ou2018optimal, @@ -1456,7 +1456,7 @@ @online{ring, title = {Ring}, url = {https://ring.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @book{rogers2000diffusions, @@ -1473,7 +1473,7 @@ year = {2018}, publisher = {TechCrunch}, howpublished = {\url{https://techcrunch.com/2018/01/28/strava-exposes-military-bases}}, - note = {Accessed on July 1, 2020} + note = {Accessed on October 11, 2021} } @inproceedings{samarati1998protecting, @@ -1667,19 +1667,19 @@ @online{tousanticovid, title = {TousAntiCovid}, url = {https://bonjour.tousanticovid.gouv.fr}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @online{transunion, title = {TransUnion}, url = {https://transunion.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @online{twitter, title = {Twitter}, url = {https://twitter.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @inproceedings{varghese2016challenges, @@ -1813,7 +1813,7 @@ @online{waze, title = {Waze}, url = {https://waze.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @incollection{wei2006time, @@ -1844,7 +1844,7 @@ @online{wiki, title = {Wikipedia}, url = {https://wikipedia.com}, - year = {Accessed on November 11, 2020} + year = {Accessed on October 11, 2021} } @misc{wired2014data, @@ -1853,7 +1853,7 @@ year = {2014}, publisher = {Wired}, howpublished = {\url{https://wired.com/insights/2014/07/data-new-oil-digital-economy}}, - note = {Accessed on December 1, 2019} + note = {Accessed on October 11, 2021} } @misc{wolfgang1999stochastic, @@ -1904,7 +1904,7 @@ @online{xssfopes2020tweet, title = {{@xssfopes: ``can anyone spot the issue with the algo? red is original data point, 400 ``anonymized'' data points calculated''}}, url = {https://twitter.com/xssfox/status/1251116087116042241}, - year = {Accessed on July 16, 2020} + year = {Accessed on October 11, 2021} } @inproceedings{yang2015bayesian, From f956e71033ec61f45ff248d4e15cd4a92b87833f Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 08:59:39 +0200 Subject: [PATCH 17/23] main: Added the french abstract and the year --- text/main.tex | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/main.tex b/text/main.tex index 020e3fd..3bb0f78 100644 --- a/text/main.tex +++ b/text/main.tex @@ -62,7 +62,7 @@ \newcommand{\thetitle}{Quality \& Privacy in User-generated Big Data: Algorithms \& Techniques} -\newcommand{\theyear}{****} +\newcommand{\theyear}{2021} \newcommand{\thedate}{***** **, \theyear} \newcommand{\thething}{landmark} @@ -85,6 +85,7 @@ \afterpage{\blankpage} \input{abstract} +\input{resume} \input{acknowledgements} \tableofcontents From 029e6b0a26ae38f7a4ddb71ba919498754c63ea6 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 09:00:22 +0200 Subject: [PATCH 18/23] evaluation: Minor corrections --- text/evaluation/summary.tex | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/evaluation/summary.tex b/text/evaluation/summary.tex index e343115..fa58627 100644 --- a/text/evaluation/summary.tex +++ b/text/evaluation/summary.tex @@ -1,10 +1,10 @@ \section{Summary} \label{sec:eval-sum} -In this chapter we presented the experimental evaluation of the {\thething} privacy mechanisms and the privacy-preserving {\thething} selection mechanism that we developed in Chapter~\ref{ch:lmdk-prv}, on real and synthetic data sets. -The Adaptive mechanism is the most reliable and best performing mechanism, in terms of overall data utility, with minimal tuning across most of the cases. +In this chapter we presented the experimental evaluation of the {\thething} privacy schemes and the privacy-preserving {\thething} selection scheme that we developed in Chapter~\ref{ch:lmdk-prv}, on real and synthetic data sets. +The Adaptive scheme is the most reliable and best performing scheme, in terms of overall data utility, with minimal tuning across most of the cases. Skip performs optimally in data sets with a smaller target value range, where approximation fits best. -The {\thething} selection mechanism introduces a reasonable data utility decline to all of our mechanisms however, the Adaptive handles it well and bounds the data utility to higher levels compared to user-level protection. +The {\thething} selection module introduces a reasonable data utility decline to all of our schemes however, the Adaptive handles it well and bounds the data utility to higher levels compared to user-level protection. % \kat{it would be nice to see it clearly on Figure 5.5. (eg, by including another bar that shows adaptive without landmark selection)} % \mk{Done.} In terms of temporal correlation, we observe that under moderate and strong temporal correlation, a greater average regular--{\thething} event distance in a {\thething} distribution causes greater overall privacy loss. -Finally, the contribution of the {\thething} privacy on enhancing the data utility, while preserving $\epsilon$-differential privacy, is demonstrated by the fact that the selected Adaptive mechanism provides better data utility than the user-level mechanism. +Finally, the contribution of the {\thething} privacy on enhancing the data utility, while preserving $\epsilon$-differential privacy, is demonstrated by the fact that the selected Adaptive scheme provides better data utility than the user-level privacy protection. From 85088d80479f183f457dfc0f02975472014b59b4 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 09:00:47 +0200 Subject: [PATCH 19/23] introduction: WIP in contribution --- text/introduction/contribution.tex | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/introduction/contribution.tex b/text/introduction/contribution.tex index 507ac8f..61a1e58 100644 --- a/text/introduction/contribution.tex +++ b/text/introduction/contribution.tex @@ -1,2 +1,4 @@ \section{Contribution} \label{sec:contr} + +\mk{WIP} From 5767899ecebf9a7078da9f9ad89ed580eeb96cb5 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 09:01:14 +0200 Subject: [PATCH 20/23] introduction: Review --- text/introduction/main.tex | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/text/introduction/main.tex b/text/introduction/main.tex index 5971fb0..0a3ce93 100644 --- a/text/introduction/main.tex +++ b/text/introduction/main.tex @@ -1,12 +1,13 @@ \chapter{Introduction} \label{ch:intro} -Data privacy is becoming an increasingly important issue both at a technical and at a societal level, and introduces various challenges ranging from the way we share and publish data sets to the way we use online and mobile services. +Data privacy is becoming an increasingly important issue, both at a technical and at a societal level, and introduces various challenges ranging from the way we share and publish data sets to the way we use online and mobile services. Personal information, also described as \emph{microdata}, acquired increasing value and are in many cases used as the `currency'~\cite{economist2016data} to pay for access to various services, i.e.,~users are asked to exchange their personal information with the service provided. -This is particularly true for many \emph{Location-Based Services} (LBSs), e.g.,~Google Maps~\cite{gmaps}, Waze~\cite{waze}, etc.; these services exchange their `free' service with collecting and using user-generated data, such as timestamped geolocalized information. -Besides navigation and location-based services, social media applications (e.g.,~Facebook~\cite{facebook}, Twitter~\cite{twitter}, Foursquare~\cite{foursquare}, etc.) take advantage of user-generated and user-related data, to make relevant recommendations and show personalized advertisement. +This is particularly true for many \emph{Location-Based Services} (LBSs), e.g.,~Google Maps~\cite{gmaps}, Waze~\cite{waze}, etc. +These services exchange their `free' service with collecting and using user-generated data, such as timestamped geolocalized information. +Besides navigation and location-based services, social media applications, e.g.,~Facebook~\cite{facebook}, Twitter~\cite{twitter}, Foursquare~\cite{foursquare}, etc. take advantage of user-generated and user-related data, to make relevant recommendations and show personalized advertisement. In this case, the location is also part of the important required personal data to be shared. -Last but not least, \emph{data brokers} (e.g.,~Experian~\cite{experian}, TransUnion~\cite{transunion}, Acxiom~\cite{acxiom}, etc.) collect data from public and private resources, e.g.,~censuses, bank card transaction records, voter registration lists, etc. +Last but not least, \emph{data brokers}, e.g.,~Experian~\cite{experian}, TransUnion~\cite{transunion}, Acxiom~\cite{acxiom}, etc. collect data from public and private resources, e.g.,~censuses, bank card transaction records, voter registration lists, etc. Most of these data are georeferenced and contain directly or indirectly location information; protecting the location of the user has become one of the most important privacy goals so far. These different sources and types of data, on the one hand give useful feedback to the involved users and/or services, and on the other hand, when combined together, provide valuable information to various internal/external analytical services. @@ -21,11 +22,11 @@ Data from crowdsourced based applications, if not protected correctly, can be ea Privacy-preserving processes usually introduce noise in the original or the aggregated data set in order to hide the sensitive information. In the case of \emph{microdata}, a privacy-protected version, containing some synthetic data as well, is generated with the intrinsic goal to make the users indistinguishable. -In the case of \emph{statistical} data (i.e.,~the results of statistical queries over the original data sets), a privacy-protected version is generated by adding noise on the actual statistical values. +In the case of \emph{statistical} data, i.e.,~the results of statistical queries over the original data sets,, a privacy-protected version is generated by adding noise on the actual statistical values. In both cases, we end up affecting the quality of the published data set. -The privacy and the utility of the `noisy' output are two contrasting desiderata, which need to be measured and balanced. -Furthermore, if we want to account for external additional information (e.g.,~linked or correlated data) and at the same time to ensure the same level of protection, we need to add additional noise, inevitably deteriorating the quality of the output. -This problem becomes particularly pertinent in the Big Data era, as the quality or \emph{Veracity} is one of the five dimensions (known as the five \emph{`V's'}) that define Big Data, and where there is an abundance of external information that cannot be ignored. +The privacy and the utility of the `noisy' output are two contrasting desiderata which need to be measured and balanced. +Furthermore, if we want to account for external additional information, e.g.,~linked or correlated data, and at the same time to ensure the same level of protection, we need to add additional noise, which inevitably deteriorates the quality of the output. +This problem becomes particularly pertinent in the Big Data era, as the quality or \emph{Veracity} is one of the five dimensions (known as the five \emph{`V's'}) that define Big Data and where there is an abundance of external information that cannot be ignored. Since this needs to be taken into account \emph{prior} to the publishing of the data set or the aggregated statistics there of, introducing external information into privacy-preserving techniques becomes part of the traditional processing flow while keeping an acceptable quality to privacy ratio. As we can observe in the examples mentioned above, there are many cases where data are not protected at source (what is also described as \emph{local} data privacy protection) for various reasons, e.g.,~the users do not want to pay extra, it is impossible due to technical complexity, because the quality of the expected service will be deteriorated, etc. @@ -37,7 +38,6 @@ Moreover, privacy-preserving algorithms are designed specifically for data publi In that respect, we need to be able to correctly choose the proper privacy algorithm(s), which would allow users to share protected copies of their data with some guarantees. The selection process is far from trivial, since it is essential to: \begin{enumerate} - \itemsep-0.25em \item select an appropriate privacy-preserving technique, relevant to the data set intended for public release; \item understand the different requirements imposed by the selected technique and tune the different parameters according to the circumstances of the use case based on, e.g.,~assumptions, level of distortion, etc.~\cite{kifer2011no}; \item get the necessary balance between privacy and data utility, which is a significant task for any privacy algorithm as well as any privacy expert. @@ -51,15 +51,13 @@ Selecting the wrong privacy algorithm or configuring it poorly may put at risk t \label{fig:data-value} \end{figure} -In data privacy research, privacy in continuous data publishing scenarios is the area that is concerned by studying the privacy problems created when sensitive data are published continuously, either infinitely (e.g.,~streaming data) or by multiple continuous publications over a known period of time (e.g.,~finite time series data). +In data privacy research, privacy in continuous data publishing scenarios is the area that is concerned by studying the privacy problems created when sensitive data are published continuously, either infinitely, e.g.,~streaming data, or by multiple continuous publications over a known period of time, e.g.,~finite time series data. This specific subfield of data privacy becomes increasingly important since it: \begin{enumerate}[(i)] - \itemsep-0.25em \item includes the most prominent cases, e.g.,~location (trajectory) privacy problems, and \item provides the most challenging and yet not well charted part of the privacy algorithms since it is rather new and increasingly complex. \end{enumerate} -In this context, we seek to offer a guide that would allow its users to choose the proper algorithm(s) for their specific use case accordingly. Additionally, data in continuous data publishing use cases require a timely processing because their value usually decreases over time depending on the use case as demonstrated in Figure~\ref{fig:data-value}. For this reason, we provide an insight into time-related properties of the algorithms, e.g.,~if they work on infinite, real-time data, or if they take into consideration existing data dependencies. The importance of continuous data publishing is stressed by the fact that, commonly, many types of data have such properties, with geospatial data being a prominent case. From f3286409a1e2258103c713c60805e4f4f2488e2e Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 09:01:34 +0200 Subject: [PATCH 21/23] Introduction: Added the structure --- text/introduction/structure.tex | 35 +++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/text/introduction/structure.tex b/text/introduction/structure.tex index 8a54fac..32300d8 100644 --- a/text/introduction/structure.tex +++ b/text/introduction/structure.tex @@ -1,2 +1,37 @@ \section{Structure} \label{sec:struct} +This thesis is structured as follows: + + +\paragraph{Chapter~\ref{ch:prel}} +introduces some relevant terminology and information around the problem of +quality and privacy in user-generated Big Data with a special focus on continuous data publishing. +First, in Section~\ref{sec:data}, we categorize user-generated data sets and review data processing in the context of continuous data publishing. +Second, in Section~\ref{sec:privacy}, we define information disclosure in data privacy. We list the categories of privacy attacks, the possible privacy protection levels, the fundamental privacy operations that are applied to achieve data privacy, and finally we provide a brief overview of the basic notions for data privacy protection. +Third, in Section~\ref{sec:correlation}, we focus on the impact of correlation on data privacy. +More particularly, we discuss the different types of correlation, we document ways to extract data correlation from continuous data, and we investigate the privacy risks that data correlation entails with special focus on the privacy loss under temporal correlation. + + +\paragraph{Chapter~\ref{ch:rel}} +reviews works that deal with privacy under continuous data publishing covering diverse use cases. +We present the relevant literature based on two levels of categorization. +First, we group works with respect to whether they deal with microdata or statistical data as input. +Then, we further group them into two subcategories depending on if they are designed for the finite or infinite observation setting. + + +\paragraph{Chapter~\ref{ch:lmdk-prv}} +proposes a novel configurable privacy scheme, \emph{{\thething} privacy} (Section~\ref{sec:thething}), which takes into account significant events (\emph{\thethings}) in the time series and allocates the available privacy budget accordingly. +We propose three privacy schemes that guarantee {\thething} privacy. +To further enhance our privacy methodology, and protect the {\thething} position in the time series, we propose techniques to perturb the initial {\thething} set (Section~\ref{sec:theotherthing}). + + +\paragraph{Chapter~\ref{ch:eval}} +presents the experiments that we performed in order to evaluate {\thething} privacy (Chapter~\ref{ch:lmdk-prv}) on real and synthetic data sets. +Section~\ref{sec:eval-dtl} contains all the details regarding the data sets the we used for our experiments along with the system configurations. +Section~\ref{sec:eval-lmdk} evaluates the data utility of the {\thething} privacy schemes that we designed in Section~\ref{sec:thething} and investigates the behavior of the privacy loss under temporal correlation for different distributions of {\thethings}. +Section~\ref{sec:eval-lmdk-sel} justifies our decisions while designing the privacy-preserving {\thething} selection module in Section~\ref{sec:theotherthing} and the data utility impact of the latter. +Finally, Section~\ref{sec:eval-sum} concludes this chapter by summarizing the main results derived from the experiments. + + +\paragraph{Chapter~\ref{ch:con}} +concludes the thesis and outlines possible future directions. From d608ada99e77715d055f068b280a18241a02bcb3 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 09:02:12 +0200 Subject: [PATCH 22/23] problem: Minor corrections --- text/problem/main.tex | 4 ++-- text/problem/thething/contribution.tex | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/text/problem/main.tex b/text/problem/main.tex index c2a1785..df35ccd 100644 --- a/text/problem/main.tex +++ b/text/problem/main.tex @@ -39,8 +39,8 @@ Event-level limits the privacy protection to \emph{any single event}, user-level \kat{Please write another introduction for your chapter, that is in connection to your thesis, not the paper.. all this information in this paragraph must be said in the introduction of the thesis, not of the chapter.. } In this chapter, we propose a novel configurable privacy scheme, \emph{\thething} privacy (Section~\ref{sec:thething}), which takes into account significant events (\emph{\thethings}) in the time series and allocates the available privacy budget accordingly. -We propose three privacy models that guarantee {\thething} privacy. -To further enhance our privacy method, and protect the {\thethings} position in the time series, we propose techniques to perturb the initial {\thethings} set (Section~\ref{sec:theotherthing}).\kat{this is the content that you must enrich and motivate more in the intro of this chapter} +We propose three privacy schemes that guarantee {\thething} privacy. +To further enhance our privacy methodology, and protect the {\thethings} position in the time series, we propose techniques to perturb the initial {\thethings} set (Section~\ref{sec:theotherthing}).\kat{this is the content that you must enrich and motivate more in the intro of this chapter} \input{problem/thething/main} \input{problem/theotherthing/main} diff --git a/text/problem/thething/contribution.tex b/text/problem/thething/contribution.tex index 8f94e04..e9870df 100644 --- a/text/problem/thething/contribution.tex +++ b/text/problem/thething/contribution.tex @@ -2,5 +2,5 @@ \label{subsec:lmdk-contrib} In this section, we formally define a novel privacy notion that we call \emph{{\thething} privacy}. -We apply this privacy notion to time series consisting of \emph{{\thethings}} and regular events, and we design and implement three {\thething} privacy mechanisms. +We apply this privacy notion to time series consisting of \emph{{\thethings}} and regular events, and we design and implement three {\thething} privacy schemes. We further study {\thething} privacy under temporal correlation that is inherent in time series publishing. From c41503ed41d8a9bd773425e9d430d4468ff45b92 Mon Sep 17 00:00:00 2001 From: Manos Date: Fri, 15 Oct 2021 09:02:44 +0200 Subject: [PATCH 23/23] related: Minor corrections --- text/related/main.tex | 4 ++-- text/related/summary.tex | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/text/related/main.tex b/text/related/main.tex index c60101a..a8f2fb7 100644 --- a/text/related/main.tex +++ b/text/related/main.tex @@ -16,8 +16,8 @@ Finally, there are some surveys on application-specific privacy challenges. For example, Zhou et al.~\cite{zhou2008brief} have a focus on social networks, and Christin et al.~\cite{christin2011survey} give an outline of how privacy aspects are addressed in crowdsensing applications. In this chapter, we document works that deal with privacy under continuous data publishing covering diverse use cases. -We present the works in the literature based on two levels of categorisation. -First, we group works with respect to whether they receive microdata or statistical data (see Section~\ref{subsec:data-categories} for the definitions) as input. +We present the works in the literature based on two levels of categorization. +First, we group works with respect to whether they deal with microdata or statistical data (see Section~\ref{subsec:data-categories} for the definitions) as input. Then, we further group them into two subcategories, whether they are designed for the finite or infinite (see Section.~\ref{subsec:data-publishing}) observation setting. \kat{continue.. say also in which category you place your work} %Such a documentation becomes very useful nowadays, due to the abundance of continuously user-generated data sets that could be analyzed and/or published in a privacy-preserving way, and the quick progress made in this research field. diff --git a/text/related/summary.tex b/text/related/summary.tex index 9f98a42..d6ca5a2 100644 --- a/text/related/summary.tex +++ b/text/related/summary.tex @@ -1,5 +1,6 @@ \section{Summary} \label{sec:sum-rel} -This is the summary of this chapter. -\kat{? Don't forget to mention here the publication that you have.} \ No newline at end of file +In this chapter, we offer a guide that would allow its users to choose the proper algorithm(s) for their specific use case. + +\kat{? Don't forget to mention here the publication that you have.}