Reading some values from multiple files raster files

Dec 17, 2010 at 8:40 AM

Which is the faster method to read raster files of equal size and produce a new one ? My files are in bgd format.

I am currently  using the following method but if i open all the files simultaneously i will end up with an out of memory exception.


    Dim Raster_1 As IRaster = Raster.OpenFile(FileName1, False, SpatialStatusStrip)

    Dim Raster_2 As IRaster = Raster.OpenFile(FileName2, False, SpatialStatusStrip)

 Dim CalcValues(NumRows, NumColumns) As Double

For iRow As Integer = 0 To NumRows - 1

           Dim RasterValues_1 As Double() = Raster_1.ToRaster(Of Double)().ReadRow(iRow)

           Dim RasterValues_2 As Double() = Raster_2.ToRaster(Of Double)().ReadRow(iRow)

           Dim RasterValues_3  As Double() = Raster_3.ToRaster(Of Double)().ReadRow(iRow)

                For iCol As Integer = 0 To NumColumns - 1

                        CalcValues(iRow, Icol) = Calc(RasterValues_1(iCol ), RasterValues_2(iCol ),  RasterValues_3(iCol ))

               Next

 Next

Developer
Dec 17, 2010 at 3:02 PM

If you know you are working with a BGD double raster, you can use the BGDRaster(Of Double) class directly and take advantage of the "ReadRaster" and "WriteRaster" methods on the class which loads only part of your raster in.  The usual threshold is to use 64 Megs or about an 8000x8000 raster, but I typically get memory exceptions if I try to load more than one raster that large into memory at a time.  The ReadRaster and WriteRaster methods I think work with a jagged array in row major format.  That means that it is an array of arrays where the outer array represents the rows and one inner array represents a column.  I can't remember exactly what a jagged array looks like in Visual Basic, but hopefully that is not unfamiliar to you.  You may still have to call GC.Collect() once each pass is out of scope or else despite being "managed" arrays, the system may not be smart enough to purge itself before starting the next batch.  If you can't seem to dimension even a new BGDRaster pointing to one of your files without it loading the content in memory, you can brute force it by doing myBgdRaster.Data = null; myBgdRaster.IsInRam = false;.  The second line may not be necessary.  Anyway, that will clear the memory that your raster is hogging by loading all the values, and then you can use the ReadRaster, WriteRaster stuff.

Ted

 

 

Dec 17, 2010 at 4:30 PM

ok Ted thanks a lot for your help

i will try these instructions and post back my new questions! 

i am guessing using short is better than double and i am pointing to the direction of using the short datatype.

anyway dotspatial seems to produce some wmi files ... what are these ?

is there a grid format that by using it dotspatial is behaving better ?

 

Developer
Dec 17, 2010 at 5:11 PM

mwi = "mapWindowImage" files.  I needed a native image ARGB image format I could store on disk with pyramids but without compression.  Compression slows things down when you need a specific sub-range of a very large image.  GDAL is way too slow for most of this kind of access, so we needed something native.  A few image formats just might be ok directly from GDAL if they have overviews, but most raster and image formats do not support overviews.  The mwi file allows you to very speedily move around a view of an image too large to fit in memory.  In the case of a raster, the extra mwi file is the actual image on your screen representing the raster.  The mwh or "MapWindowHeader" file is the xml header for the mwi file and shows the row and column offsets for the overviews.  I think the bytes are in ARGB order and can very easily be converted into a bitmap that supports alpha transparency.  These tend to be monster files, but they can be safely deleted when the image is not in use.  We leave it around because they are frequently slow to create, so we might as well use it if it already exists.  The only drawback is that it wants to be written to the same place you are loading an image, which could be a no-no on a CD or in someones very sensitive shared database system.  It is a good idea to copy your data to an editable location before opening it with DotSpatial.  This is a good idea with any open source app because there could be bugs or crashes that may compromise your original, and that would be bad.

Ted

 

 

Developer
Dec 17, 2010 at 5:12 PM

P.S. you should be able to open, create or work with them using the "PyramidImage" class.

Ted