| --- |
| license: apache-2.0 |
| --- |
| # Prototype v12 - Patch 16 - 128x128 images - 1.2m imagenet images |
|
|
| Well the 64x64 image set worked just fine, so it's time to upgrade and test the limits of the architecture. |
|
|
| Can it simply... scale? ooooor do we need more solvers along the way to compensate? |
|
|
| benjamin-paine/imagenet-1k-128x128 |
|
|
| Can we actually solve it |
|
|
| YES WE DID! |
|
|
|  |
|
|
| It seems geometric manifolds learn... differently than standard manifolds, don't they. |
|
|
|
|
| # Prototype V11 - Patch16 - MSE 0.0005 - 64x64 tiny imagenet |
|
|
| I'd say it works. |
|
|
| The images show it works. It works. |
|
|
|
|
|  |
|
|
|
|
| ``` |
| Using geolip-core SVD (fp64 Gram+eigh (FL=available, N<=12)) |
| PatchSVAE - 16 patches of 16×16 |
| Dataset: tiny_imagenet (64×64, 200 classes) |
| Per-patch: (256, 16) = 4096 elements, rows on S^15 |
| Encoder/Decoder: hidden=768, depth=4 (residual blocks) |
| Cross-attention: 2 layers on S vectors (2,272 params) |
| Soft hand: boost=1.5x near CV=0.125, penalty=0.3 far |
| Total params: 16,942,419 |
| =============================================================================================== |
| ep | loss recon t/ep | t_rec | S0 SD ratio erank | row_cv prox rw | S_delta |
| ----------------------------------------------------------------------------------------------- |
| 1 | 0.2595 0.1806 12.2 | 0.1024 | 5.036 3.254 1.55 15.87 | 0.2007 0.905 1.45 | 0.09694 a:0.0242/0.0247 |
| 2 | 0.1216 0.0845 12.3 | 0.0675 | 5.071 3.298 1.54 15.88 | 0.2018 0.885 1.44 | 0.17411 a:0.0251/0.0257 |
| 3 | 0.0847 0.0587 12.3 | 0.0470 | 5.093 3.312 1.54 15.88 | 0.2046 0.869 1.43 | 0.19894 a:0.0258/0.0265 |
| 4 | 0.0623 0.0432 12.3 | 0.0430 | 5.115 3.323 1.54 15.88 | 0.2006 0.864 1.43 | 0.20848 a:0.0264/0.0272 |
| 6 | 0.0359 0.0248 12.3 | 0.0198 | 5.129 3.332 1.54 15.88 | 0.2006 0.907 1.45 | 0.21832 a:0.0273/0.0281 |
| 8 | 0.0225 0.0155 12.2 | 0.0196 | 5.149 3.341 1.54 15.87 | 0.2017 0.876 1.44 | 0.22351 a:0.0279/0.0287 |
| 10 | 0.0170 0.0116 12.3 | 0.0100 | 5.151 3.352 1.54 15.88 | 0.2035 0.924 1.46 | 0.22671 a:0.0283/0.0290 |
| 12 | 0.0141 0.0096 12.3 | 0.0114 | 5.159 3.354 1.54 15.88 | 0.2009 0.909 1.45 | 0.22924 a:0.0285/0.0293 |
| 14 | 0.0121 0.0082 12.3 | 0.0073 | 5.156 3.362 1.53 15.88 | 0.2018 0.855 1.43 | 0.23137 a:0.0288/0.0296 |
| 16 | 0.0105 0.0072 12.3 | 0.0108 | 5.161 3.363 1.53 15.88 | 0.2003 0.860 1.43 | 0.23316 a:0.0290/0.0298 |
| 18 | 0.0094 0.0064 12.3 | 0.0055 | 5.158 3.365 1.53 15.88 | 0.2017 0.879 1.44 | 0.23467 a:0.0292/0.0300 |
| 20 | 0.0086 0.0058 12.3 | 0.0050 | 5.157 3.367 1.53 15.88 | 0.2023 0.805 1.40 | 0.23601 a:0.0293/0.0301 |
| 22 | 0.0079 0.0054 12.4 | 0.0045 | 5.157 3.369 1.53 15.88 | 0.1996 0.872 1.44 | 0.23726 a:0.0295/0.0303 |
| 24 | 0.0074 0.0050 12.2 | 0.0064 | 5.146 3.380 1.52 15.88 | 0.2044 0.879 1.44 | 0.23848 a:0.0296/0.0305 |
| 26 | 0.0068 0.0046 12.4 | 0.0039 | 5.155 3.372 1.53 15.88 | 0.2036 0.884 1.44 | 0.23955 a:0.0297/0.0306 |
| 28 | 0.0063 0.0042 12.3 | 0.0036 | 5.155 3.378 1.53 15.88 | 0.2077 0.841 1.42 | 0.24057 a:0.0299/0.0307 |
| 30 | 0.0058 0.0038 12.3 | 0.0038 | 5.155 3.380 1.53 15.88 | 0.2027 0.911 1.46 | 0.24149 a:0.0300/0.0309 |
| 32 | 0.0055 0.0036 12.2 | 0.0032 | 5.150 3.383 1.52 15.88 | 0.2045 0.807 1.40 | 0.24239 a:0.0301/0.0310 |
| 34 | 0.0054 0.0036 12.3 | 0.0037 | 5.145 3.388 1.52 15.88 | 0.1996 0.875 1.44 | 0.24329 a:0.0302/0.0311 |
| 36 | 0.0049 0.0032 12.3 | 0.0031 | 5.154 3.385 1.52 15.88 | 0.2054 0.828 1.41 | 0.24409 a:0.0303/0.0312 |
| 38 | 0.0046 0.0030 12.3 | 0.0027 | 5.152 3.390 1.52 15.88 | 0.2038 0.847 1.42 | 0.24490 a:0.0304/0.0313 |
| 40 | 0.0044 0.0029 12.3 | 0.0032 | 5.155 3.392 1.52 15.89 | 0.2046 0.855 1.43 | 0.24566 a:0.0305/0.0314 |
| 42 | 0.0043 0.0028 12.3 | 0.0024 | 5.152 3.395 1.52 15.89 | 0.2064 0.905 1.45 | 0.24637 a:0.0305/0.0315 |
| 44 | 0.0042 0.0027 12.3 | 0.0023 | 5.150 3.395 1.52 15.89 | 0.2084 0.844 1.42 | 0.24705 a:0.0306/0.0316 |
| 46 | 0.0039 0.0025 12.3 | 0.0022 | 5.149 3.400 1.51 15.89 | 0.2057 0.868 1.43 | 0.24776 a:0.0307/0.0317 |
| 48 | 0.0037 0.0024 12.3 | 0.0024 | 5.152 3.403 1.51 15.89 | 0.2138 0.831 1.42 | 0.24843 a:0.0308/0.0318 |
| 50 | 0.0038 0.0024 12.3 | 0.0025 | 5.149 3.406 1.51 15.89 | 0.2078 0.810 1.40 | 0.24906 a:0.0309/0.0319 |
| 52 | 0.0034 0.0021 12.3 | 0.0019 | 5.154 3.405 1.51 15.89 | 0.2082 0.872 1.44 | 0.24965 a:0.0309/0.0320 |
| 54 | 0.0033 0.0020 12.2 | 0.0019 | 5.156 3.406 1.51 15.89 | 0.2085 0.894 1.45 | 0.25022 a:0.0310/0.0320 |
| 56 | 0.0033 0.0020 12.4 | 0.0019 | 5.150 3.412 1.51 15.89 | 0.2058 0.866 1.43 | 0.25079 a:0.0311/0.0321 |
| 58 | 0.0031 0.0019 12.3 | 0.0033 | 5.147 3.416 1.51 15.89 | 0.2071 0.774 1.39 | 0.25135 a:0.0311/0.0322 |
| 60 | 0.0030 0.0018 12.4 | 0.0017 | 5.153 3.415 1.51 15.89 | 0.2134 0.840 1.42 | 0.25187 a:0.0312/0.0323 |
| 62 | 0.0030 0.0018 12.3 | 0.0016 | 5.155 3.416 1.51 15.89 | 0.2080 0.764 1.38 | 0.25235 a:0.0313/0.0323 |
| 64 | 0.0028 0.0017 12.2 | 0.0014 | 5.156 3.416 1.51 15.89 | 0.2100 0.666 1.33 | 0.25285 a:0.0313/0.0324 |
| 66 | 0.0028 0.0017 12.3 | 0.0017 | 5.151 3.419 1.51 15.89 | 0.2101 0.865 1.43 | 0.25333 a:0.0314/0.0324 |
| 68 | 0.0026 0.0015 12.3 | 0.0014 | 5.158 3.419 1.51 15.89 | 0.2078 0.838 1.42 | 0.25381 a:0.0314/0.0325 |
| 70 | 0.0025 0.0015 12.4 | 0.0021 | 5.160 3.422 1.51 15.89 | 0.2112 0.806 1.40 | 0.25428 a:0.0315/0.0326 |
| 72 | 0.0026 0.0015 12.2 | 0.0013 | 5.158 3.422 1.51 15.89 | 0.2126 0.835 1.42 | 0.25471 a:0.0316/0.0326 |
| 74 | 0.0024 0.0014 12.2 | 0.0015 | 5.154 3.427 1.50 15.89 | 0.2143 0.838 1.42 | 0.25514 a:0.0316/0.0327 |
| 76 | 0.0024 0.0014 12.2 | 0.0012 | 5.161 3.424 1.51 15.89 | 0.2151 0.847 1.42 | 0.25553 a:0.0317/0.0327 |
| 78 | 0.0023 0.0013 12.3 | 0.0014 | 5.157 3.428 1.50 15.89 | 0.2121 0.686 1.34 | 0.25592 a:0.0317/0.0328 |
| 80 | 0.0024 0.0013 12.3 | 0.0012 | 5.160 3.428 1.51 15.89 | 0.2068 0.824 1.41 | 0.25630 a:0.0317/0.0328 |
| 82 | 0.0027 0.0016 12.2 | 0.0015 | 5.146 3.394 1.52 15.89 | 0.2065 0.899 1.45 | 0.25687 a:0.0318/0.0329 |
| 84 | 0.0022 0.0013 12.2 | 0.0013 | 5.156 3.405 1.51 15.89 | 0.2092 0.875 1.44 | 0.25709 a:0.0319/0.0329 |
| 86 | 0.0022 0.0012 12.3 | 0.0022 | 5.154 3.413 1.51 15.89 | 0.2091 0.835 1.42 | 0.25726 a:0.0319/0.0329 |
| 88 | 0.0021 0.0012 12.3 | 0.0014 | 5.154 3.417 1.51 15.89 | 0.2074 0.840 1.42 | 0.25740 a:0.0319/0.0329 |
| 90 | 0.0022 0.0013 12.3 | 0.0030 | 5.147 3.416 1.51 15.89 | 0.2191 0.848 1.42 | 0.25753 a:0.0319/0.0329 |
| 92 | 0.0021 0.0011 12.3 | 0.0012 | 5.157 3.418 1.51 15.89 | 0.2123 0.775 1.39 | 0.25766 a:0.0319/0.0329 |
| 94 | 0.0021 0.0011 12.3 | 0.0010 | 5.156 3.419 1.51 15.89 | 0.2117 0.710 1.35 | 0.25779 a:0.0319/0.0329 |
| 96 | 0.0020 0.0011 12.3 | 0.0013 | 5.154 3.420 1.51 15.89 | 0.2166 0.940 1.47 | 0.25793 a:0.0319/0.0330 |
| 98 | 0.0019 0.0011 12.2 | 0.0010 | 5.156 3.421 1.51 15.89 | 0.2143 0.762 1.38 | 0.25807 a:0.0320/0.0330 |
| 100 | 0.0020 0.0010 12.3 | 0.0009 | 5.155 3.422 1.51 15.89 | 0.2173 0.642 1.32 | 0.25821 a:0.0320/0.0330 |
| 102 | 0.0020 0.0010 12.2 | 0.0009 | 5.156 3.423 1.51 15.89 | 0.2165 0.868 1.43 | 0.25835 a:0.0320/0.0330 |
| 104 | 0.0019 0.0010 12.4 | 0.0009 | 5.157 3.423 1.51 15.89 | 0.2125 0.788 1.39 | 0.25850 a:0.0320/0.0330 |
| 106 | 0.0019 0.0009 12.3 | 0.0009 | 5.156 3.424 1.51 15.89 | 0.2219 0.666 1.33 | 0.25866 a:0.0320/0.0331 |
| 108 | 0.0019 0.0009 12.3 | 0.0009 | 5.153 3.425 1.50 15.89 | 0.2202 0.671 1.34 | 0.25881 a:0.0321/0.0331 |
| 110 | 0.0020 0.0009 12.3 | 0.0011 | 5.153 3.427 1.50 15.89 | 0.2163 0.726 1.36 | 0.25896 a:0.0321/0.0331 |
| 112 | 0.0019 0.0009 12.4 | 0.0009 | 5.155 3.427 1.50 15.89 | 0.2205 0.837 1.42 | 0.25911 a:0.0321/0.0331 |
| 114 | 0.0019 0.0008 12.3 | 0.0008 | 5.155 3.427 1.50 15.89 | 0.2220 0.803 1.40 | 0.25926 a:0.0321/0.0332 |
| 116 | 0.0018 0.0008 12.3 | 0.0009 | 5.155 3.427 1.50 15.89 | 0.2211 0.852 1.43 | 0.25942 a:0.0321/0.0332 |
| 118 | 0.0019 0.0008 12.3 | 0.0008 | 5.153 3.429 1.50 15.89 | 0.2207 0.694 1.35 | 0.25957 a:0.0321/0.0332 |
| 120 | 0.0018 0.0008 12.2 | 0.0008 | 5.156 3.429 1.50 15.89 | 0.2271 0.664 1.33 | 0.25972 a:0.0322/0.0332 |
| 122 | 0.0018 0.0008 12.3 | 0.0008 | 5.154 3.429 1.50 15.89 | 0.2266 0.658 1.33 | 0.25986 a:0.0322/0.0333 |
| 124 | 0.0017 0.0007 12.3 | 0.0008 | 5.154 3.431 1.50 15.89 | 0.2201 0.771 1.39 | 0.26000 a:0.0322/0.0333 |
| 126 | 0.0017 0.0007 12.4 | 0.0008 | 5.157 3.430 1.50 15.89 | 0.2253 0.862 1.43 | 0.26014 a:0.0322/0.0333 |
| 128 | 0.0018 0.0007 12.3 | 0.0008 | 5.153 3.431 1.50 15.89 | 0.2222 0.638 1.32 | 0.26027 a:0.0322/0.0333 |
| 130 | 0.0018 0.0007 12.3 | 0.0007 | 5.155 3.431 1.50 15.89 | 0.2255 0.786 1.39 | 0.26040 a:0.0322/0.0333 |
| 132 | 0.0018 0.0007 12.3 | 0.0007 | 5.153 3.432 1.50 15.89 | 0.2325 0.778 1.39 | 0.26053 a:0.0323/0.0333 |
| 134 | 0.0017 0.0007 12.3 | 0.0007 | 5.154 3.433 1.50 15.89 | 0.2250 0.876 1.44 | 0.26065 a:0.0323/0.0334 |
| 136 | 0.0017 0.0006 12.3 | 0.0007 | 5.157 3.432 1.50 15.89 | 0.2269 0.866 1.43 | 0.26077 a:0.0323/0.0334 |
| 138 | 0.0017 0.0006 12.3 | 0.0007 | 5.156 3.433 1.50 15.89 | 0.2247 0.760 1.38 | 0.26088 a:0.0323/0.0334 |
| 140 | 0.0016 0.0006 12.3 | 0.0006 | 5.157 3.433 1.50 15.89 | 0.2242 0.725 1.36 | 0.26099 a:0.0323/0.0334 |
| 142 | 0.0016 0.0006 12.3 | 0.0006 | 5.157 3.433 1.50 15.89 | 0.2241 0.909 1.45 | 0.26109 a:0.0323/0.0334 |
| 144 | 0.0017 0.0006 12.3 | 0.0006 | 5.157 3.433 1.50 15.89 | 0.2287 0.815 1.41 | 0.26119 a:0.0323/0.0334 |
| 146 | 0.0016 0.0006 12.3 | 0.0007 | 5.158 3.434 1.50 15.90 | 0.2205 0.722 1.36 | 0.26128 a:0.0324/0.0334 |
| 148 | 0.0016 0.0006 12.3 | 0.0008 | 5.157 3.434 1.50 15.90 | 0.2286 0.691 1.35 | 0.26137 a:0.0324/0.0335 |
| 150 | 0.0016 0.0006 12.3 | 0.0006 | 5.158 3.434 1.50 15.90 | 0.2259 0.845 1.42 | 0.26146 a:0.0324/0.0335 |
| 152 | 0.0017 0.0006 12.3 | 0.0006 | 5.158 3.434 1.50 15.90 | 0.2295 0.757 1.38 | 0.26154 a:0.0324/0.0335 |
| 154 | 0.0016 0.0005 12.3 | 0.0006 | 5.159 3.435 1.50 15.90 | 0.2304 0.751 1.38 | 0.26162 a:0.0324/0.0335 |
| 156 | 0.0018 0.0005 12.3 | 0.0006 | 5.159 3.435 1.50 15.90 | 0.2264 0.796 1.40 | 0.26169 a:0.0324/0.0335 |
| 158 | 0.0017 0.0005 12.3 | 0.0006 | 5.160 3.434 1.50 15.90 | 0.2282 0.788 1.39 | 0.26176 a:0.0324/0.0335 |
| 160 | 0.0017 0.0005 12.3 | 0.0005 | 5.161 3.434 1.50 15.90 | 0.2291 0.766 1.38 | 0.26183 a:0.0324/0.0335 |
| 162 | 0.0016 0.0005 12.3 | 0.0005 | 5.161 3.434 1.50 15.90 | 0.2282 0.716 1.36 | 0.26189 a:0.0324/0.0335 |
| 164 | 0.0016 0.0005 12.3 | 0.0005 | 5.161 3.435 1.50 15.90 | 0.2344 0.792 1.40 | 0.26196 a:0.0324/0.0335 |
| 166 | 0.0016 0.0005 12.3 | 0.0006 | 5.162 3.434 1.50 15.90 | 0.2305 0.707 1.35 | 0.26202 a:0.0324/0.0335 |
| 168 | 0.0016 0.0005 12.3 | 0.0005 | 5.162 3.434 1.50 15.90 | 0.2353 0.816 1.41 | 0.26207 a:0.0324/0.0335 |
| 170 | 0.0016 0.0005 12.3 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2296 0.756 1.38 | 0.26213 a:0.0325/0.0335 |
| 172 | 0.0018 0.0005 12.4 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2391 0.742 1.37 | 0.26218 a:0.0325/0.0336 |
| 174 | 0.0016 0.0005 12.4 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2307 0.863 1.43 | 0.26224 a:0.0325/0.0336 |
| 176 | 0.0016 0.0005 12.3 | 0.0005 | 5.163 3.434 1.50 15.90 | 0.2329 0.854 1.43 | 0.26228 a:0.0325/0.0336 |
| 178 | 0.0017 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2287 0.803 1.40 | 0.26233 a:0.0325/0.0336 |
| 180 | 0.0017 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2361 0.819 1.41 | 0.26237 a:0.0325/0.0336 |
| 182 | 0.0018 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2360 0.729 1.36 | 0.26241 a:0.0325/0.0336 |
| 184 | 0.0016 0.0005 12.3 | 0.0005 | 5.164 3.434 1.50 15.90 | 0.2395 0.774 1.39 | 0.26245 a:0.0325/0.0336 |
| 186 | 0.0015 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2325 0.777 1.39 | 0.26248 a:0.0325/0.0336 |
| 188 | 0.0018 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2430 0.669 1.33 | 0.26250 a:0.0325/0.0336 |
| 190 | 0.0016 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2293 0.781 1.39 | 0.26252 a:0.0325/0.0336 |
| 192 | 0.0017 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2246 0.763 1.38 | 0.26254 a:0.0325/0.0336 |
| 194 | 0.0019 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2324 0.764 1.38 | 0.26255 a:0.0325/0.0336 |
| 196 | 0.0016 0.0005 12.2 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2301 0.868 1.43 | 0.26256 a:0.0325/0.0336 |
| 198 | 0.0016 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2332 0.696 1.35 | 0.26256 a:0.0325/0.0336 |
| 200 | 0.0017 0.0005 12.3 | 0.0005 | 5.165 3.434 1.50 15.90 | 0.2397 0.808 1.40 | 0.26256 a:0.0325/0.0336 |
| |
| ========================================================================================== |
| FINAL ANALYSIS |
| ========================================================================================== |
| |
| PatchSVAE: 16 patches × (256, 16) |
| Target CV: 0.125 |
| Recon MSE: 0.000489 +/- 0.000743 |
| Row CV: 0.2397 |
| Cross-attention S delta: 0.26256 |
| |
| Learned alpha per mode (coordination strength): |
| Layer 0: mean=0.0327 max=0.0336 min=0.0323 |
| α[ 0]: 0.0324 ###################################### |
| α[ 1]: 0.0323 ###################################### |
| α[ 2]: 0.0327 ###################################### |
| α[ 3]: 0.0326 ###################################### |
| α[ 4]: 0.0325 ###################################### |
| α[ 5]: 0.0326 ###################################### |
| α[ 6]: 0.0332 ####################################### |
| α[ 7]: 0.0336 ####################################### |
| α[ 8]: 0.0326 ###################################### |
| α[ 9]: 0.0324 ###################################### |
| α[10]: 0.0326 ###################################### |
| α[11]: 0.0325 ###################################### |
| α[12]: 0.0328 ####################################### |
| α[13]: 0.0324 ###################################### |
| α[14]: 0.0331 ####################################### |
| α[15]: 0.0326 ###################################### |
| Layer 1: mean=0.0323 max=0.0327 min=0.0315 |
| α[ 0]: 0.0324 ####################################### |
| α[ 1]: 0.0326 ####################################### |
| α[ 2]: 0.0323 ####################################### |
| α[ 3]: 0.0326 ####################################### |
| α[ 4]: 0.0327 ####################################### |
| α[ 5]: 0.0326 ####################################### |
| α[ 6]: 0.0320 ####################################### |
| α[ 7]: 0.0315 ###################################### |
| α[ 8]: 0.0324 ####################################### |
| α[ 9]: 0.0327 ####################################### |
| α[10]: 0.0325 ####################################### |
| α[11]: 0.0324 ####################################### |
| α[12]: 0.0321 ####################################### |
| α[13]: 0.0324 ####################################### |
| α[14]: 0.0317 ###################################### |
| α[15]: 0.0322 ####################################### |
| |
| Coordinated singular value profile: |
| S[ 0]: 5.1650 cum= 9.2% ############################# |
| S[ 1]: 4.9525 cum= 17.6% ############################ |
| S[ 2]: 4.8142 cum= 25.6% ########################### |
| S[ 3]: 4.6335 cum= 32.9% ########################## |
| S[ 4]: 4.5199 cum= 40.0% ########################## |
| S[ 5]: 4.4203 cum= 46.7% ######################### |
| S[ 6]: 4.3376 cum= 53.2% ######################### |
| S[ 7]: 4.2448 cum= 59.3% ######################## |
| S[ 8]: 4.1641 cum= 65.3% ######################## |
| S[ 9]: 4.0915 cum= 71.1% ####################### |
| S[10]: 4.0086 cum= 76.6% ####################### |
| S[11]: 3.9144 cum= 81.9% ###################### |
| S[12]: 3.7995 cum= 86.8% ###################### |
| S[13]: 3.6926 cum= 91.5% ##################### |
| S[14]: 3.5949 cum= 95.9% #################### |
| S[15]: 3.4336 cum=100.0% ################### |
| |
| Saving reconstruction grid... |
| Saved to /content/svae_patch_recon.png |
| ``` |
|
|
|
|
| # Prototype V10.3 - Patch16 - The VIT size. |
|
|
| Might need some tweaks but I don't think so. We're approaching actual vit prototype accuracy now. |
|
|
| Lets see how the SVAE performs. |
|
|
| # Prototype V10.2 - Patch32 - Patchwork Cross-Attention with Edge Smoothing |
|
|
| This eliminates the edge cutting of the last version, and in the process the recon accuracy has gone up. |
|
|
| Model still escapes the discharge within 2 epochs and has robust recon. |
|
|
| Defeated the last version. |
|  |
|
|
|
|
|  |
|
|
| # Prototype V10.1 Patchwork Cross-Attention - Stabilized |
|
|
| The patchwork has stabilized, and the output is more accurate than the original now that it supports SVD 32 with more accuracy and higher speed |
|
|
| Epoch 28 hit the unstable point, but the gradient clipped attention was the ticket that ensured solidity. |
|
|
| The discharge recovered immediately. |
|
|
|  |
|
|
|
|
| Give or take 97% accurate recall, lets get those numbers up before we move onto more powerful image sets. Roughly 28m params. |
|
|
|  |
|
|
| ``` |
| 174 | 0.0471 0.0315 8.0 | 0.0318 | 4.038 2.092 1.93 31.52 | 0.1271 0.995 1.50 | 0.26864 a:0.0471/0.0476 |
| 176 | 0.0471 0.0314 7.9 | 0.0318 | 4.038 2.092 1.93 31.52 | 0.1343 1.000 1.50 | 0.26874 a:0.0471/0.0476 |
| 178 | 0.0471 0.0314 7.9 | 0.0318 | 4.038 2.093 1.93 31.52 | 0.1313 0.995 1.50 | 0.26883 a:0.0471/0.0477 |
| 180 | 0.0471 0.0314 8.0 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1312 1.000 1.50 | 0.26892 a:0.0471/0.0477 |
| 182 | 0.0471 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1310 0.995 1.50 | 0.26899 a:0.0471/0.0477 |
| 184 | 0.0471 0.0314 7.9 | 0.0317 | 4.038 2.092 1.93 31.52 | 0.1350 0.993 1.50 | 0.26906 a:0.0472/0.0477 |
| 186 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.092 1.93 31.52 | 0.1338 1.000 1.50 | 0.26911 a:0.0472/0.0477 |
| 188 | 0.0470 0.0314 8.0 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1305 0.999 1.50 | 0.26916 a:0.0472/0.0477 |
| 190 | 0.0470 0.0314 8.0 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1358 0.999 1.50 | 0.26919 a:0.0472/0.0477 |
| 192 | 0.0470 0.0314 8.0 | 0.0317 | 4.038 2.092 1.93 31.52 | 0.1354 0.999 1.50 | 0.26922 a:0.0472/0.0477 |
| 194 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1301 0.992 1.50 | 0.26923 a:0.0472/0.0477 |
| 196 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1330 0.998 1.50 | 0.26924 a:0.0472/0.0477 |
| 198 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1312 1.000 1.50 | 0.26925 a:0.0472/0.0477 |
| 200 | 0.0470 0.0314 7.9 | 0.0317 | 4.038 2.093 1.93 31.52 | 0.1300 1.000 1.50 | 0.26925 a:0.0472/0.0477 |
| |
| ========================================================================================== |
| FINAL ANALYSIS |
| ========================================================================================== |
| |
| PatchSVAE: 4 patches × (256, 32) |
| Target CV: 0.125 |
| Recon MSE: 0.031701 +/- 0.024789 |
| Row CV: 0.1300 |
| Cross-attention S delta: 0.26925 |
| |
| Learned alpha per mode (coordination strength): |
| Layer 0: mean=0.0471 max=0.0477 min=0.0466 |
| α[ 0]: 0.0470 ####################################### |
| α[ 1]: 0.0473 ####################################### |
| α[ 2]: 0.0474 ####################################### |
| α[ 3]: 0.0473 ####################################### |
| α[ 4]: 0.0471 ####################################### |
| α[ 5]: 0.0474 ####################################### |
| α[ 6]: 0.0469 ####################################### |
| α[ 7]: 0.0472 ####################################### |
| α[ 8]: 0.0470 ####################################### |
| α[ 9]: 0.0475 ####################################### |
| α[10]: 0.0467 ####################################### |
| α[11]: 0.0471 ####################################### |
| α[12]: 0.0477 ####################################### |
| α[13]: 0.0466 ####################################### |
| α[14]: 0.0471 ####################################### |
| α[15]: 0.0472 ####################################### |
| α[16]: 0.0472 ####################################### |
| α[17]: 0.0471 ####################################### |
| α[18]: 0.0470 ####################################### |
| α[19]: 0.0475 ####################################### |
| α[20]: 0.0466 ####################################### |
| α[21]: 0.0477 ####################################### |
| α[22]: 0.0470 ####################################### |
| α[23]: 0.0469 ####################################### |
| α[24]: 0.0472 ####################################### |
| α[25]: 0.0472 ####################################### |
| α[26]: 0.0471 ####################################### |
| α[27]: 0.0471 ####################################### |
| α[28]: 0.0474 ####################################### |
| α[29]: 0.0472 ####################################### |
| α[30]: 0.0466 ####################################### |
| α[31]: 0.0475 ####################################### |
| Layer 1: mean=0.0472 max=0.0477 min=0.0466 |
| α[ 0]: 0.0474 ####################################### |
| α[ 1]: 0.0472 ####################################### |
| α[ 2]: 0.0470 ####################################### |
| α[ 3]: 0.0471 ####################################### |
| α[ 4]: 0.0474 ####################################### |
| α[ 5]: 0.0470 ####################################### |
| α[ 6]: 0.0474 ####################################### |
| α[ 7]: 0.0473 ####################################### |
| α[ 8]: 0.0473 ####################################### |
| α[ 9]: 0.0470 ####################################### |
| α[10]: 0.0477 ####################################### |
| α[11]: 0.0472 ####################################### |
| α[12]: 0.0466 ####################################### |
| α[13]: 0.0477 ####################################### |
| α[14]: 0.0473 ####################################### |
| α[15]: 0.0471 ####################################### |
| α[16]: 0.0472 ####################################### |
| α[17]: 0.0471 ####################################### |
| α[18]: 0.0476 ####################################### |
| α[19]: 0.0470 ####################################### |
| α[20]: 0.0475 ####################################### |
| α[21]: 0.0470 ####################################### |
| α[22]: 0.0472 ####################################### |
| α[23]: 0.0475 ####################################### |
| α[24]: 0.0472 ####################################### |
| α[25]: 0.0471 ####################################### |
| α[26]: 0.0475 ####################################### |
| α[27]: 0.0474 ####################################### |
| α[28]: 0.0472 ####################################### |
| α[29]: 0.0469 ####################################### |
| α[30]: 0.0476 ####################################### |
| α[31]: 0.0466 ####################################### |
| |
| Coordinated singular value profile: |
| S[ 0]: 4.0376 cum= 5.3% ############################# |
| S[ 1]: 3.9321 cum= 10.3% ############################# |
| S[ 2]: 3.8501 cum= 15.1% ############################ |
| S[ 3]: 3.7785 cum= 19.8% ############################ |
| S[ 4]: 3.7092 cum= 24.2% ########################### |
| S[ 5]: 3.6414 cum= 28.5% ########################### |
| S[ 6]: 3.5771 cum= 32.7% ########################## |
| S[ 7]: 3.5158 cum= 36.7% ########################## |
| S[ 8]: 3.4554 cum= 40.6% ######################### |
| S[ 9]: 3.3961 cum= 44.3% ######################### |
| S[10]: 3.3371 cum= 48.0% ######################## |
| S[11]: 3.2788 cum= 51.5% ######################## |
| S[12]: 3.2230 cum= 54.8% ####################### |
| S[13]: 3.1681 cum= 58.1% ####################### |
| S[14]: 3.1141 cum= 61.2% ####################### |
| S[15]: 3.0607 cum= 64.3% ###################### |
| S[16]: 3.0088 cum= 67.2% ###################### |
| S[17]: 2.9568 cum= 70.1% ##################### |
| S[18]: 2.9075 cum= 72.8% ##################### |
| S[19]: 2.8572 cum= 75.5% ##################### |
| S[20]: 2.8067 cum= 78.0% #################### |
| S[21]: 2.7584 cum= 80.5% #################### |
| S[22]: 2.7075 cum= 82.9% #################### |
| S[23]: 2.6574 cum= 85.2% ################### |
| S[24]: 2.6060 cum= 87.4% ################### |
| S[25]: 2.5535 cum= 89.5% ################## |
| S[26]: 2.4991 cum= 91.5% ################## |
| S[27]: 2.4413 cum= 93.5% ################## |
| S[28]: 2.3770 cum= 95.3% ################# |
| S[29]: 2.2906 cum= 97.0% ################# |
| S[30]: 2.2012 cum= 98.6% ################ |
| S[31]: 2.0926 cum=100.0% ############### |
| |
| Saving reconstruction grid... |
| Saved to /content/svae_patch_recon.png |
| ``` |
|
|
|
|
| # Prototype V10 Patchwork Cross-Attention - Unstable |
|
|
| Tiny Imagenet can't draw enough information from a single monotonic MLP projection, so I'm breaking the structure into quadrant-based mlp patches |
| with cross-attention for a prototype. |
|
|
| Each patch is 32x32 and they have svd 24 independently represented each with patchwork cross-attention. Similar to a vit, |
| so I'm building it to a full vit structure over time to ensure solidity and solidarity. |
|
|
|  |
|
|
| Current proto is more stable but requires a bit more oomph. |
|
|
| The CV is enjoying it's drift a BIT too much |
|
|
| I'll try attention alpha rather than rigid alpha. 4 patches is a bit unstable, so lets get some stability. |
|
|
|
|
| # Prototype V9 prod |
|
|
| Should run on colab. Install the necessary repos. |
|
|
| https://huggingface.co/AbstractPhil/geolip-SVAE/blob/main/prototype_v9_prod.py |
|
|
|
|
| # Prototype V8 Soft Hand Loss |
|
|
| Stable prototype found. Scaling with the CV ratio within this band is a stable attractor to the structural response. |
|
|
| The soft hand loss is acting like a stable attractant. Correct utilization of this behavior can directly attenuate |
| a model's structural internals to align to certain trajectory-based routes. |
|
|
| The alignment can be directly tuned at runtime, shifted to learn implicit rules, altered to teach specific behaviors, and more. |
|
|
| 0.034 mse, which is a different gauge of loss entirely. |
|  |
|
|
|
|
| # Prototype V7 |
|
|
| Normalized spherical without magnitude, expected considerably faster with less accuracy at first stages. |
|
|
|  |
|
|
|
|
| # What happens if you train with the wrong CV value? |
|
|
| ``` |
| Using geolip-core SVD (Gram + eigh) |
| SVAE - V=96, D=24 (Validated: CV=0.3668) |
| Matrix: (96, 24) = 2304 elements |
| SVD: geolip-core Gram+eigh |
| Losses: recon + CV(w=0.1, target=0.3668) |
| Params: 6,036,736 |
| ===================================================================================== |
| ep | loss recon cv_l t/ep | t_rec | S0 SD ratio erank | row_cv |
| ------------------------------------------------------------------------------------- |
| 1 | 0.4174 0.4169 0.0037 7.3 | 0.2843 | 5.39 1.977 2.73 23.15 | 0.3039 |
| 2 | 0.2492 0.2489 0.0031 7.3 | 0.2286 | 5.43 1.978 2.75 23.14 | 0.3148 |
| 3 | 0.2096 0.2093 0.0030 7.3 | 0.1946 | 5.50 1.982 2.77 23.13 | 0.3352 |
| 4 | 0.1858 0.1855 0.0021 7.2 | 0.1812 | 5.48 1.980 2.77 23.13 | 0.3460 |
| 6 | 0.1586 0.1581 0.0046 7.3 | 0.1541 | 5.31 1.873 2.83 23.09 | 0.3938 |
| 8 | 0.1419 0.1407 0.0096 7.3 | 0.1377 | 5.33 1.815 2.93 23.03 | 0.4565 |
| 10 | 0.1314 0.1283 0.0385 7.3 | 0.1279 | 5.42 1.778 3.05 22.97 | 0.5373 |
| 12 | 0.1226 0.1160 0.0599 7.2 | 0.1162 | 5.67 1.738 3.26 22.86 | 0.6060 |
| 14 | 0.1189 0.1087 0.0847 7.1 | 0.1109 | 5.78 1.705 3.39 22.79 | 0.6643 |
| 16 | 0.1175 0.1014 0.1935 7.1 | 0.0996 | 6.17 1.701 3.63 22.67 | 0.7598 |
| 18 | 0.1170 0.0952 0.2238 7.2 | 0.0974 | 6.50 1.671 3.89 22.52 | 0.8211 |
| 20 | 0.1173 0.0905 0.1539 7.2 | 0.0907 | 6.69 1.649 4.06 22.43 | 0.8383 |
| 22 | 0.1200 0.0852 0.3335 7.1 | 0.0903 | 7.11 1.655 4.30 22.30 | 0.9128 |
| 24 | 0.1233 0.0817 0.2770 7.2 | 0.0831 | 7.51 1.646 4.56 22.15 | 0.9654 |
| 26 | 0.1286 0.0785 0.3243 7.1 | 0.0778 | 7.71 1.646 4.68 22.09 | 1.0196 |
| 28 | 0.1328 0.0752 0.4244 7.2 | 0.0780 | 7.84 1.636 4.80 22.02 | 1.1002 |
| 30 | 0.1373 0.0726 0.8786 7.1 | 0.0752 | 8.24 1.631 5.05 21.87 | 1.1243 |
| 32 | 0.1437 0.0703 0.6946 7.2 | 0.0704 | 8.52 1.631 5.23 21.76 | 1.2061 |
| 34 | 0.6025 0.6020 0.0062 7.1 | 0.5194 | 28.25 10.261 2.75 23.14 | 0.2935 |
| 36 | 0.4995 0.4990 0.0062 7.2 | 0.4949 | 29.82 10.939 2.73 23.15 | 0.2982 |
| 38 | 0.4947 0.4942 0.0058 7.2 | 0.4915 | 28.37 10.433 2.72 23.15 | 0.2988 |
| 40 | 0.4579 0.4574 0.0053 7.2 | 0.4557 | 26.31 9.585 2.74 23.14 | 0.3041 |
| 42 | 0.4333 0.4328 0.0051 7.1 | 0.4259 | 22.03 7.996 2.75 23.14 | 0.2984 |
| 44 | 0.4057 0.4054 0.0038 7.1 | 0.3880 | 21.15 7.656 2.76 23.15 | 0.3177 |
| 46 | 0.3670 0.3667 0.0024 7.2 | 0.3634 | 19.33 6.943 2.78 23.13 | 0.3280 |
| 48 | 0.3495 0.3493 0.0005 7.1 | 0.3457 | 18.34 6.569 2.79 23.13 | 0.3336 |
| 50 | 0.3341 0.3340 0.0010 7.3 | 0.3326 | 17.55 6.298 2.79 23.13 | 0.3424 |
| 52 | 0.3205 0.3204 0.0003 7.2 | 0.3182 | 16.94 6.069 2.79 23.12 | 0.3549 |
| ``` |
|
|
| SNAP. right there at epoch 34. The tension was too strong, the model simply snapped. |
| I had it set to around 0.366, and it requires that value there where it snapped to. 0.2935 |
|
|
| The actual value as of the bulk embedding tests show; CV=0.2992 is the stable attractor, almost precisely where the model snapped to. |
|
|
| The effect was so strong, that the entire model had a forced reset when it realized the fundamental invalidity. |
|
|
| Why? I don't know yet. |
|
|
| # Models |
|
|
| V4 5m SVD+EIGH 100 epochs 48x24 |
|
|
|  |
|
|
| V4 111m 200x24 SVD+EIGH KL_DIV - Undercooked, needs more epochs -> sequel faulty, collapse |
| |
|  |
| |
| |
| V3 v1024 - SVD 24 |
| |
| |
|  |
| |
| V2 16 modes |
| |
|  |
| |
| |
| |
| V1 8 modes |
|  |
| |
