GeoSANE — generating geospatial models from weights

The running example asks one initialization to combine radar-relevant geospatial structure, dense-prediction structure, and a Swin-B backbone. Yet the paper describes remote-sensing models as specialists across sensors, resolutions, and objectives, with complementary rather than unified capabilities. ^[1][1] Introduction“However, despite their success, the landscape of remote sensing foundation models (RSFMs) remains fragmented: each model specializes in a subset of sensors, spatial resolutions, or objectives, with…”Open source No single checkpoint is therefore guaranteed to cover all three.

The obvious move is model selection: find the checkpoint closest to the flood task and fine-tune it. That preserves one model’s complete parameterization, but it also accepts that model’s particular trade-off. A radar specialist can use a different architecture; a Swin-B checkpoint can lack radar specialization. Because the collection’s capabilities are complementary rather than unified, selecting one endpoint cannot combine information distributed across several specialists. ^[1][1] Introduction“However, despite their success, the landscape of remote sensing foundation models (RSFMs) remains fragmented: each model specializes in a subset of sensors, spatial resolutions, or objectives, with…”Open source

Direct merging looks like the next obvious attempt. If two checkpoints share aligned parameter slots, their weights can be combined in parameter space. But heterogeneous checkpoints do not provide that alignment: a Swin block, a convolutional block, and a task head divide computation into different tensors. The merging methods discussed by the paper generally assume a shared architecture and initialization, precisely the constraints this collection violates. ^[2][2] Related Work“Nonetheless, most of these techniques assume that all models share the same architecture and initialization. In contrast, GeoSANE is designed to encapsulate the knowledge of a large collection of…”Open source Forcing an average would therefore confuse structural incompatibility with knowledge transfer.

GeoSANE changes what counts as the dataset. Its filtered collection contains 103 remote-sensing models with about 38 billion parameters, spanning foundation and task-specific checkpoints. ^[3][3] Remote Sensing Model Collection“After filtering, the final collection contains 103 (foundation) remote sensing models, representing approximately 38 billion parameters.”Open source A weight corpus is this population treated as training examples rather than as candidates from which one winner must be chosen. The corpus supplies inputs for learning a shared representation of neural-network weights. ^[4][4] Remote Sensing Model Collection“This model collection serves as the training dataset for GeoSANE, i.e., as input to learn a shared latent representation of neural network weights.”Open source Because the representation is learned across the population, GeoSANE is designed to capture relationships among models without requiring them to share an architecture or initialization. ^[2][2] Related Work“Nonetheless, most of these techniques assume that all models share the same architecture and initialization. In contrast, GeoSANE is designed to encapsulate the knowledge of a large collection of…”Open source

That reframes the Swin-B prompt: it can specify the requested backbone while the learned weight representation supplies information drawn from a broader geospatial population. But calling checkpoints a corpus does not make their raw tensors uniform. The collection includes ViTs, Swins, ResNets, UNets, detectors, and multimodal models; these expose different layer shapes and parameter counts. ^[5][5] Remote Sensing Model Collection“GeoSANE is designed to automatically load and process a wide variety of architectures, including Transformer-based backbones (ViT, Swin, etc.), CNNs (ResNet, UNet, MobileNet, etc.), multimodal rada…”Open source They still cannot enter one sequence model directly.

Take a Swin-B checkpoint: flattening its layer tensors would produce a scalar sequence, but the stream would no longer mark where one layer ends and the next begins. Checkpoints with different parameter counts would also produce different lengths. GeoSANE therefore needs uniform token vectors that retain layer provenance and support fixed-size batches; its sequence representation handles models of different lengths, architectures, and sizes. ^[6][6] Learning the Shared Latent Representation“The core idea of~\cite{Schrholt2024TowardsSA} is to tokenize model weights and express an entire model as a sequence of token vectors. Using such a configuration allows the encoder-decoder backbone…”Open source

GeoSANE first reshapes each layer’s weights into a two-dimensional matrix, keeping the layer as the unit of organization. It then splits those matrices into fixed-width vectors. If a matrix portion is too large, splitting produces several vectors; if the final portion is too short, zero padding fills the unused coordinates. Concatenating the vectors produces the weight token sequence $\mathbf{T}$. Fixed width standardizes each token, while splitting lets the full sequence remain as long as the checkpoint requires. ^[7][7] Learning the Shared Latent Representation“To that end, the weights $\mathbf{w}$ of models in the model collection are loaded and reshaped into 2D matrices per layer, then divided into fixed-size tokens $\mathbf{T}_n$ of size $d_t$. Zero pa…”Open source

Fixed width alone does not recover provenance. Each token therefore receives positional metadata recording its absolute sequence position, layer index, and position within that layer. For the Swin-B prompt, two equal-looking vectors from different attention blocks remain distinguishable because their metadata differs. ^[8][8] Learning the Shared Latent Representation“Each token is augmented with a 3D positional embedding $\mathbf{P}=[n,l,k]$ indicating absolute sequence position $n$, layer index $l$, and within-layer position $k$.”Open source

Models still produce sequences of different lengths, so GeoSANE divides each sequence into fixed-length chunks called windows. Windows make batch dimensions agree without requiring a ResNet, ViT, and Swin checkpoint to contain the same parameter count. Processing chunks also avoids forcing an entire large checkpoint through the encoder-decoder at once; the sequence can be reconstructed window by window. ^[9][9] Learning the Shared Latent Representation“For training, the tokenized model weights are divided into fixed-length chunks of a specified size (referred to as window) to allow uniform batch sizes and to efficiently process architectures of v…”Open source

Padding now creates a bookkeeping hazard. A padded zero has the same numeric value as a genuine learned parameter equal to zero. If both entered reconstruction error, the decoder would be trained to reproduce storage introduced by tokenization rather than only model weights.

The parameter mask $\mathbf{M}$ resolves that ambiguity. Its value is $1$ at every real parameter coordinate and $0$ at every padded coordinate. Reconstruction differences are multiplied by this mask, so changing a prediction at padding contributes exactly nothing to the loss; changing a prediction at a genuine zero-valued parameter still contributes normally. Padding standardizes shape, and the mask compensates by removing those added coordinates from the training target. ^[10][10] Learning the Shared Latent Representation“Here, the mask $\mathbf{M}$ is used to separate real parameters with $1$ from padding with $0$, ensuring that the loss is only computed on actual weights.”Open source

Combined with per-layer tokenization, positional metadata, and fixed-length windows, this yields a unified sequence format while retaining layer and within-layer provenance. ^[7][7] Learning the Shared Latent Representation“To that end, the weights $\mathbf{w}$ of models in the model collection are loaded and reshaped into 2D matrices per layer, then divided into fixed-size tokens $\mathbf{T}_n$ of size $d_t$. Zero pa…”Open source ^[8][8] Learning the Shared Latent Representation“Each token is augmented with a 3D positional embedding $\mathbf{P}=[n,l,k]$ indicating absolute sequence position $n$, layer index $l$, and within-layer position $k$.”Open source ^[9][9] Learning the Shared Latent Representation“For training, the tokenized model weights are divided into fixed-length chunks of a specified size (referred to as window) to allow uniform batch sizes and to efficiently process architectures of v…”Open source But reconstruction alone can be minimized by copying each input. It does not determine how two views of the same model or checkpoints from different models should be arranged in latent space.

Suppose the tokenized Swin-B checkpoint is reconstructed almost perfectly, window by window. That sounds sufficient, but the encoder could merely preserve enough local detail to copy each chunk. Nothing in that task requires two views of the same checkpoint to occupy nearby regions of latent representation, or checkpoints from different models to remain distinct. Accurate copying alone therefore leaves the model-level geometry unspecified. ^[11][11] Learning the Shared Latent Representation“The contrastive term uses two augmented views $i,j$ of the same model: the first is the original token sequence, while the second is a noised version of it. Both views are processed through the enc…”Open source

reconstruction loss supplies the copying pressure. The decoder reconstructs each real weight token, and the squared difference from the original contributes to the error. parameter mask multiplies padded positions by $0$, so reducing error on invented padding cannot improve the objective; only actual checkpoint parameters count. ^[10][10] Learning the Shared Latent Representation“Here, the mask $\mathbf{M}$ is used to separate real parameters with $1$ from padding with $0$, ensuring that the loss is only computed on actual weights.”Open source

contrastive loss supplies the missing model-level pressure. GeoSANE creates two views of one model—the original token sequence and a noised copy—then passes both through the encoder and projection head. The NT-Xent criterion, a batch contrastive loss, pulls their projected embeddings together while keeping them distinct from embeddings of other models. Noise can alter local values, but model identity should survive. ^[11][11] Learning the Shared Latent Representation“The contrastive term uses two augmented views $i,j$ of the same model: the first is the original token sequence, while the second is a noised version of it. Both views are processed through the enc…”Open source

The balance matters. At $\gamma=0$, training rewards parameter fidelity but provides no contrastive force organizing checkpoints. At $\gamma=1$, the objective ignores reconstruction, so nearby embeddings need not decode into faithful Swin-B weight chunks. Intermediate values ask latent representation to retain decodable parameters while making model identity stable under noise. GeoSANE is trained with this combined reconstruction and contrastive guidance rather than either term alone. ^[13][13] Implementation Details“The model is optimized using a combination of reconstruction loss and contrastive guidance, as described in Section~\ref{sec:methods}.”Open source

The two losses pressure the latent space to preserve real parameters and organize models at once. But the objective leaves a concrete problem unresolved: how does generation select a region whose decoded weights match the requested Swin-B architecture?

For Sen1Floods11, sampling anywhere in the shared latent space has no Swin-B-specific anchor: the decoder might produce a sequence associated with another tensor layout. The obvious safe alternative is to copy the ImageNet-pretrained Swin-B prompt, but that preserves both its architecture and its existing initialization. One attempt risks incompatible weights; the other generates nothing new. GeoSANE needs the prompt to constrain structure without forcing every parameter to remain equal to the prompt.

GeoSANE first tokenizes prompt-model weights from the ImageNet-pretrained Swin-B and passes the resulting weight token sequence through the weight encoder. The prompt is therefore not an example to copy after generation. It supplies an architectural reference and a location from which generation can begin. ^[14][14] Generating new Models from the Latent Space“Given a \textit{prompt model} $a$ (e.g., an ImageNet-pretrained ViT-L or Swin-B from the \texttt{timm} library~\cite{rw2019timm}), we tokenize and encode its weights $\mathbf{w}_a$ into the latent…”Open source

The encoder output is prompt latent representation, the prompt’s address in the learned weight space. GeoSANE fits a kernel density estimator around that address. A kernel density estimator, or KDE, builds a local probability distribution from nearby latent values rather than assuming one fixed parametric shape. GeoSANE then samples from this local distribution. Locality is the compromise: every sample remains near the prompt’s structurally similar region, but it need not equal the prompt’s latent representation. The sampled point can therefore carry a different parameter configuration shaped by the geospatial model collection used to train the latent space. ^[15][15] Generating new Models from the Latent Space“We then fit a Kernel Density Estimator (KDE) around $\mathbf{Z}_a$ and draw samples $\tilde{\mathbf{z}}$ from this local distribution. This sampling procedure explores nearby regions in the latent…”Open source

The weight decoder maps each sampled latent point to candidate weight tokens. Detokenization then places those token values back into the tensor layout specified by the Swin-B prompt, producing generated model weights. The compensating constraint is explicit: parameter values change, while the architecture—including its layer arrangement and tensor structure—remains that of the prompt. GeoSANE can generate several such candidates because sampling and decoding require only forward passes; the paper describes selecting candidates by a simple performance criterion before fine-tuning. ^[16][16] Generating new Models from the Latent Space“Each sampled latent representation $\tilde{\mathbf{z}}$ is decoded using the decoder to produce synthetic weight tokens $\tilde{\mathbf{T}} = h_{\psi}(\tilde{\mathbf{z}})$, which are subsequently d…”Open source ^[17][17] Generating new Models from the Latent Space“Sampling in latent space is inexpensive, as both sampling and decoding require only forward passes. This makes it feasible to generate multiple candidate models and select top-$m$ candidates accord…”Open source

The resulting backbone can be loaded into the Sen1Floods11 segmentation pipeline, but loading proves only structural compatibility. It does not show that the new initialization contains useful geospatial structure, beats the unchanged ImageNet prompt, or improves on random initialization. Those claims require the same Swin-B architecture and equal fine-tuning conditions for all three starting points.

In the Sen1Floods11 example, GeoSANE-generated weights and the ImageNet-pretrained prompt are evaluated with Swin-B backbones under the same fine-tuning budget. ^[18][18] Performance of Generated Models“Comparison between finetuning GeoSANE-generated models vs. direct finetuning of the models used as prompts for GeoSANE (ImageNet-pretrained ViT-L for classification and Swin-B for segmentation and…”Open source A strong score therefore leaves several explanations open: Swin-B may simply fit the task well, ImageNet pretraining may supply useful features, or downstream training may erase differences between starting weights. Comparing only against random initialization is the obvious test, but it cannot separate those alternatives.

The first control locks the architecture and training duration. GeoSANE-generated models are compared with randomly initialized models of identical architecture, fine-tuned for the same number of epochs under similar conditions. This asks whether the generated initialization contributes anything beyond Swin-B and ordinary task training. ^[19][19] Performance of Generated Models“Table \ref{tab:scratch_vs_geosane} compares models generated by GeoSANE with randomly initialized ones of identical architectures and finetuned for the same number of epochs, under similar conditions.”Open source The experiments fine-tune generated models for $50$ epochs with AdamW and select the final checkpoint by validation loss. ^[20][20] Performance of Generated Models“For downstream evaluation, all generated models are fine-tuned for 50 epochs\footnote{We finetune all models for 50 epochs for fair comparison, although performance usually saturates earlier (Fig~\…”Open source A second control starts from the unchanged ImageNet-pretrained Swin-B prompt and gives it the same $50$-epoch fine-tuning budget as the generated Swin-B. ^[18][18] Performance of Generated Models“Comparison between finetuning GeoSANE-generated models vs. direct finetuning of the models used as prompts for GeoSANE (ImageNet-pretrained ViT-L for classification and Swin-B for segmentation and…”Open source That comparison is stricter: architecture and prompt pretraining are now shared, while the initial parameter values differ.

Selection needs separate accounting. GeoSANE can generate several candidates and retain the top candidates using a performance criterion before fine-tuning. ^[17][17] Generating new Models from the Latent Space“Sampling in latent space is inexpensive, as both sampling and decoding require only forward passes. This makes it feasible to generate multiple candidate models and select top-$m$ candidates accord…”Open source If scratch and prompt baselines do not receive an equivalent selection opportunity, the comparison measures the complete generation-and-selection procedure, not only geospatial structure in the sampled weights. The matched controls still answer a useful operational question—whether that procedure supplies a better starting point—but they do not assign every gain to one causal component.

The remaining comparisons attack different substitutes. Existing remote-sensing foundation models test whether generated weights compete with conventional large-scale pretraining. ^[21][21] Performance of Generated Models“For classification tasks we use ViT-L backbones, and for segmentation and detection tasks we use Swin-B. As shown in Table \ref{tab:sota_comparison}, GeoSANE achieves the best or second-best result…”Open source DARE merges pairs of foundation models and is compared with a GeoSANE model of the same architecture, testing direct parameter combination as an alternative explanation. ^[22][22] Performance of Generated Models“We therefore merge pairs of remote sensing foundation models using the DARE (\textbf{D}rop \textbf{A}nd \textbf{RE}scale)~\cite{Yu2023LanguageMA} method and compare the resulting merged models to G…”Open source Lightweight experiments compare generated ResNet-18 and MobileNetV2 weights with magnitude pruning, variational dropout, and knowledge distillation. The ResNet-18 comparisons use $11$ million parameters; for MobileNetV2, pruning retains about $5$ million nonzero parameters, while GeoSANE and distillation use $3.5$ million. ^[23][23] Downstream Tasks and Datasets“Comparison of GeoSANE with pruning and distillation baselines. For \textit{Magnitude Pruning} and \textit{Variational Dropout}, we prune pretrained Remote Sensing Foundation Models (RSFMs) (ScaleMA…”Open source These experiments cover classification, segmentation, and object detection. ^[24][24] Downstream Tasks and Datasets“We evaluate models generated by GeoSANE on a diverse set of downstream tasks, covering classification, segmentation, and object detection across multiple modalities.”Open source Gains across these controls support GeoSANE as a useful initialization procedure, but exceptions and ablations without remote-sensing weights determine how far that conclusion extends.

The generated Swin-B improves flood segmentation, but one score cannot reveal whether the gain is broad or merely a favorable task. ^[25][25] Performance of Generated Models“Sen1Floods11 & 85.2 & \textbf{89.6} &\small \textcolor{green!60!black}{+4.4} \\”Open source Even a broad leaderboard leaves a second ambiguity: generation might preserve generic computer-vision structure without learning anything specific to geospatial models. The evidence needs both matched controls and an intervention on the weight corpus.

The matched scratch comparison holds architecture and fine-tuning duration constant, removing the obvious explanation that GeoSANE simply received a stronger backbone or more optimization. Under those conditions, generated initializations outperform scratch across all ten reported datasets. ^[19][19] Performance of Generated Models“Table \ref{tab:scratch_vs_geosane} compares models generated by GeoSANE with randomly initialized ones of identical architectures and finetuned for the same number of epochs, under similar conditions.”Open source ^[26][26] Performance of Generated Models“Across ten diverse datasets, GeoSANE consistently outperforms training from scratch, with particularly large gains on more challenging, heterogeneous datasets such as fMoW and BigEarthNet, which co…”Open source In the running example, direct fine-tuning of the ImageNet-pretrained Swin-B prompt scores $85.2$ on Sen1Floods11, while the generated initialization scores $89.6$ under the same $50$-epoch budget. ^[25][25] Performance of Generated Models“Sen1Floods11 & 85.2 & \textbf{89.6} &\small \textcolor{green!60!black}{+4.4} \\”Open source That establishes an initialization gain beyond both random weights and the unchanged prompt; it does not yet identify where that gain came from.

The remote-sensing ablation supplies the missing intervention. GeoSANE first generates from a latent space trained only on general computer-vision weights, then from the same system after training on remote-sensing weights; downstream fine-tuning is identical. The computer-vision-only outputs perform similarly to their ImageNet prompts, whereas adding remote-sensing weights improves every reported task, including Sen1Floods11 from $86.0$ to $89.6$. ^[27][27] Effect of Remote Sensing Fine-tuning on GeoSANE Model Generation“When comparing Table~\ref{tab:cv_vs_rs_finetune} with the results in Table~\ref{tab:anchor-model} (models prompt vs. GeoSANE-generated models), we observe that models generated without remote sensi…”Open source Architecture-compatible generation alone therefore does little beyond the prompt. The improvement appears when geospatial checkpoints change the learned weight space.

The opening claim therefore closes narrowly. GeoSANE uses a heterogeneous remote-sensing model collection as training data for a shared weight representation. ^[4][4] Remote Sensing Model Collection“This model collection serves as the training dataset for GeoSANE, i.e., as input to learn a shared latent representation of neural network weights.”Open source Under matched architecture and fine-tuning conditions, its generated initializations beat scratch across ten datasets, and adding remote-sensing weights yields gains beyond computer-vision-only generation. ^[19][19] Performance of Generated Models“Table \ref{tab:scratch_vs_geosane} compares models generated by GeoSANE with randomly initialized ones of identical architectures and finetuned for the same number of epochs, under similar conditions.”Open source ^[26][26] Performance of Generated Models“Across ten diverse datasets, GeoSANE consistently outperforms training from scratch, with particularly large gains on more challenging, heterogeneous datasets such as fMoW and BigEarthNet, which co…”Open source ^[27][27] Effect of Remote Sensing Fine-tuning on GeoSANE Model Generation“When comparing Table~\ref{tab:cv_vs_rs_finetune} with the results in Table~\ref{tab:anchor-model} (models prompt vs. GeoSANE-generated models), we observe that models generated without remote sensi…”Open source These experiments establish reuse of knowledge carried by existing checkpoints; they do not establish that data pretraining can be removed from the larger ecosystem.