This project is read-only.

large raster datasets

Jan 13, 2011 at 9:59 PM

Any guestimate on the current limits for raster dataset size that can be visualized effectively?

Related question: what are the plans for supporting tiled/pyrmaid raster datasets?

The use-case I'm looking at would use something similar to NASA's blue marble imagery as a background raster layer, draw some vector layers on top of it -- lat/lon lines, political boundaries, airport locations. Zooming in would result in higher resolution raster data being drawn, and additional vector layers appearing (roads/rivers, finer resolution lat/lon lines, etc).

How close to the above could I get with the current implementaion? It looks like I can control a layer's visibility based on zoom level... that might be good enough for now.

Thanks,

Steve

Jan 14, 2011 at 1:29 PM

Dear Steve, I have the same problem and I cannot load an ecw image up to 100MB.

Ted, when could be available this option?

Thanks in advanced.

Francisco J.

Jan 14, 2011 at 4:32 PM

I noticed this comment on http://mapwindow6.codeplex.com/

Don't know if it will help you...

Big rasters/grids (> 8000x8000) fail to load or use unconventional native formats

  • (requires finishing img(HFA) support for native, pyramidal img files)

 

Jan 17, 2011 at 7:29 PM
mudnug wrote:

I noticed this comment on http://mapwindow6.codeplex.com/

Don't know if it will help you...

Big rasters/grids (> 8000x8000) fail to load or use unconventional native formats

  • (requires finishing img(HFA) support for native, pyramidal img files)

 

 Any rough estimate on how much work (both time estimate and $) would be involved in finishing the large raster support?

In another thread, the 'code for cash' topic was brought up... Need to confer w/ my product manager to verify my company would be open to funding the work, but that might be an option.

-Steve

Jan 17, 2011 at 7:54 PM

I think that the > 8000 x 8000 case should be supported (though admittedly somewhat slow) if you have GDAL.  What should be happening is that it creates a file-based pyramid image MapWindow Image (mwi) file.  It should do this regardless of whether you are looking at an image or a raster, though in the case of the raster, the mwi file changes each time you update the symbology. The program will need write permission on the drive where the images are stored, which may be an issue.  There is not really a solution for the idea of needing to create a file however, since GDAL seems kind of slow to try to use for image navigation in many cases (especially in cases like png which compress the entire image so to look at any small part of the image, the whole image must be decompressed).  These files are too large to store in memory which means they have to go on disk somewhere.  What we could do is add a feature to allow the user to specify where these temporary files are created, rather than always using the same location as the source.  We could also add an option to automatically delete the temporary file when finished.  I chose to leave them around, however, so that we could directly open that without having to rebuild the pyramids again each time.  Also, certain formats have built in overviews that are directly accessible through GDAL.  (ECW, J2K, img and possibly a few others).  For these formats, it might be doable to write code that directly accesses the images in GDAL and does not create a pyramid image, but would take some time to tweak all the areas where a pyramid image is expected and not found.  (lots of breakage expected when adding that feature).  Also, we could conceivably pick a native popular format like img to work with.  That is, support C# managed code to work directly with the image formats.  This would likely improve performance for that format, and would give people an option that doesn't require GDAL but which could be read by other systems.  Right now the only native grids we support are bgd, which I don't think are accepted anywhere outside of the world of MapWindow.

If the DemoMap application is not currently able to handle some large images or if memory issues are coming up at this stage, it is no longer tied to the original 8000 x 8000 limitation of in-ram image size, it is a separate bug.  So basically, the software is currently designed to support large rasters and images.  If it isn't working, then it's probably a bug that needs fixing.

Ted

 

Jan 17, 2011 at 9:05 PM

If it can't write an image pyramid to the folder containing the original, using IO.Path.GetTempPath seems like a reasonable choice. Maybe this path could be a property of something and it would default to GetTempPath if not set by the application. Actually prompting the user for a location as we try to open an image would not be very user-friendly if it becomes the stupid dialog that ALWAYS pops up.

Jan 17, 2011 at 9:18 PM

Yes, I agree.  Maybe an application setting.  I haven't decided where that goes yet.  For a final application you would just add it to the list of properties in the application\Properties\Settings file.  However, I'm not sure whether this type of thing should belong to a specific component?  The AppManager might be a place to put them, but that is tantamount to requiring the extra stuff of an AppManager.  I'll have to think about it, but yes, we definitely need to figure out the best place to store some basic user and application settings.

Ted

 

Jan 18, 2011 at 7:39 PM
shade1974 wrote:

If the DemoMap application is not currently able to handle some large images or if memory issues are coming up at this stage, it is no longer tied to the original 8000 x 8000 limitation of in-ram image size, it is a separate bug.  So basically, the software is currently designed to support large rasters and images.  If it isn't working, then it's probably a bug that needs fixing.

Ted 

Thanks for the info Ted. I'd come across that message mentioning the 8k x 8k limit before and was concerned, but hadn't actually tried it.

FWIW, I've donwloaded some big images from http://www.naturalearthdata.com/ and loaded them in DemoMap (x64, .Net 4.0). I've got a 680MB tiff image as the background and 8 Shapefile vector layers (1:10mill roads, railways, countries, states, rivers, lakes, minor islands, populated places also from naturalearthdata.com).

On my beefy system (8 core CPU, 16 GB RAM), performance is OK but not great -- 2+ GB mem usage and I can pan around the map with some lag but not too bad. I haven't tried it yet, but I assume if I tiled up the background image performance would improve.

 

Thanks,

Steve

Feb 16, 2011 at 7:33 AM
Edited Feb 16, 2011 at 7:37 AM

Friends, I follow having problem with ECW images, because I got memory exception with a 250 MB image.

 

How can I do to load this images, ECW and MrSID. For MrSID image, in a 85 MB image I got a 2,5 GB pyramid file and in the last of the process i get an error. (Se ha intentado mover el puntero más allá del archivo), I suppose that is for memory problem. A question, is needed this memory ammount?

 

For ECW images, I cannot get it only with some small images (below 50 MB of size).

I use the last release (12/02/2011) and use the DemoMap.exe and a custom application following the instructions that you give me.

 

I upload a sample of an ECW file (250 MB aproximately).

http://www.megaupload.com/?d=M9WT0U7V

 

Note.

I work with a Pentium IV 3,5 Ghz with 4GB Ram and 500 GB harddisk.

 

Thanks in advanced.

Francisco J.

Feb 16, 2011 at 9:53 PM

Hi Francisco,

I wanted to make sure the 1.8 GDAL binaries weren't the culprit.  When I was doing testing earlier, I noticed the memory problem using GDAL 1.7 as well as with GDAL 1.8.  I also downloaded your ECW image and tried it in a non DotSpatial application that used the same GDAL 1.8 and it handled it fine.  So, I think it is something with how Dot Spatial does imagery and is an old problem.  That is probably old news for most folks based on earlier posts, but I wanted to make sure nothing bad got introduced with 1.8.

Kyle

Feb 16, 2011 at 10:16 PM

Thanks for checking that Kyle.  The pyramid image theoretically should allow these to be viewed without a memory problem, but when you start talking about gigabytes it's possible that there is some other kind of memory issue cropping up.   For the various wavelet formats, we could also try to explore getting overviews directly from the GDAL image, rather than creating a pyramid image.  The problem I have had in the past is that reading bands through GDAL seems to be slow in many cases.  But maybe we need an optional setting that simply doesn't ever use a pyramid at all.  Let the program use whatever overviews exist, but if none exist, then it just uses the base image and parses the image as fast as whatever GDAL will give us and don't worry about it.

Wavelet technology applies absolutely fantastic levels of compression.  Having a 2.5 GB pyramid image represents the size of the uncompressed original image, which is probably more than 1.5 GB as well as the overviews, which probably add an additional 1GB.  It would not surprise me at all to learn that wavelet compression reduced the source image to 85 Megs.  What is less obvious is why there is a memory problem.  Francisco, thanks for posting the sample dataset.  I will see if I can track down the issue tonight.

Ted

 

Feb 17, 2011 at 7:41 AM

Many thanks to both Kile and Ted for your reply.

I will wait for any sugestion to load this image.

I got the same error when I load several jpeg images (up to 5 or 6 images), each of these are 10 MB size aproximately, because I thought that if I cannot load the ECW I could load all images, but when I loaded up to 5 or 6 images, the program crash.

Thanks in advanced.

 

Francisco J.

Feb 23, 2011 at 5:14 PM
Edited Mar 31, 2011 at 12:30 PM

Dear friends,

Has someone tried loading the image that I uploaded to the server?

Thanks in advanced.
Francisco J.