Loading heavy shapefile data

Mar 2, 2011 at 3:48 AM

Hi,

I tried to load heavy shapefile data around 2.5GB to an application which developed by dotspatial API. Observed that the application is not loading the data and cause memory exception. The same data I loaded in MapWindow4 application and successfully loaded within 20-30 seconds.

Is there any solution to this issue?. Since Dotspatial is higher version of Mapwindow it is supposed to show better performance.

Please reply.

Shabeer

Developer
Mar 2, 2011 at 2:17 PM

Hello Shabeer,

Currently, DotSpatial layers are RAM based.  All of the feature data is brought into memory.  This allows it to be very fast displaying, but has the problem of running out of memory for large data sets.

Ted and I were working on a file-based scheme.  This would result in a slightly slower display, but a small memory footprint.  We got some of the lower level pieces working, but never got around to implementing the Map Layer to use the file-based approach. I did not have enough DotSpatial system architecture knowledge to finish it.  I was able to get around it for my own use by implementing my own Map Layers that have just the functionality that my application needs, but it is not anywhere nearly as functional as is required for full-blown Dot Spatial Map use.

Some of the lower level pieces we got working implement the IFeatureSource and IShapeSource interfaces which is what I use in my own file-based map layers.  I think it would be a wonderful improvement to Dot Spatial if this work could be finished.  It would allow Dot Spatial to be used in more demanding situations and thus get more widespread use.  It is going to take someone like you or me who really needs this capability and has some good coding skills to jump in and learn and finish the job.  If you are interested in helping with this, I will send you all the information I know about this issue.

Thanks,

Kyle

Mar 4, 2011 at 3:55 AM

I am interested 

 

cdzhanglei@gmail.com

Mar 9, 2011 at 12:55 AM

Hi,

I want a good component like mapwindow ocx , that can load heavy files and display. But Dotspatial do not have this capability. Is there any plan to make it happen?. For the time being I can proceed with Mapwindow4 application. I decided to checkout the code and modify with my requirement. I need to know which revision is more stable and effort is minimum.

I heard that development of Mapwindow4 is stopped (not sure). In case is that may I get technical support from Mapwindow4 development team?

Thanks

shabeer

Mar 31, 2011 at 5:07 PM

This problem is bigger than it looks.

A shapefile with 400 Mbytes of size, it throws a "outofmemory" exception.... it's a problem for use of DotSpatial in real cases. Mapwindow can load without problem.

I'm reading the code source right now, but it a bit "complicated" to follow the loading process to improve it.

Developer
Mar 31, 2011 at 6:19 PM

I have created an issue for this:

http://dotspatial.codeplex.com/workitem/277

I attached a document to the issue that contains all the information I could find when Ted, others, and I were looking at this issue before.  See the issue and the attachment for more details.

Thanks,

Kyle

Coordinator
Apr 19, 2011 at 2:18 PM

MapWindow 4 is still being developed and the ActiveX control continues to be improved here: mapwingis.codeplex.com. It has about 10 years head start on DotSpatial, so we are still trying to bring DotSpatial up to the same performance level with these large shapefiles. - Dan

Editor
Sep 21, 2011 at 12:47 PM
kellison wrote:
Ted and I were working on a file-based scheme.  This would result in a slightly slower display, but a small memory footprint.  We got some of the lower level pieces working, but never got around to implementing the Map Layer to use the file-based approach. I did not have enough DotSpatial system architecture knowledge to finish it.  I was able to get around it for my own use by implementing my own Map Layers that have just the functionality that my application needs, but it is not anywhere nearly as functional as is required for full-blown Dot Spatial Map use.

Has there been any progress on this?

Cheers FObermaier

Developer
Sep 21, 2011 at 1:12 PM

Nope.  I had to implement my own simplified Map Layers that make use of the lower level shapefile I/O components (file-based) that Ted and I implemented.  What is lacking are Map Layers that play well with the rest of the DotSpatial system.

Kyle

Sep 22, 2011 at 3:44 PM

I am not sure I understand why DotSpatial would use memory residing data as compared to file residing. Speed has been mentioned but if the map can only handle small sized layers what s there to gain. Real world GIS applications use large datasets. If a GIS application cannot handle these the application gets overlooked and users move on to another.

Developer
Sep 23, 2011 at 6:28 PM

I wasn't on the project in the beginning, but my take is that they were able to get something up and running fairly quickly with the memory based approach that would work for the intially targeted vertical applications which did not have large dataset requirements.  Sometimes you have to walk before you can run.  I have put a lot of effort into performance tuning the new low-level shapefile I/O components and am reasonably happy with the speed I am getting.  I am confident and hopeful we can get there eventually.  We just need somebody to dive in and implement new map layers that use the file-based components.  If I had more time, I'd give it a go, but just can't do it right now.

Kyle

Sep 23, 2011 at 6:37 PM

Makes sense. Thank-you for the explanation. Sounds like things are going in the right direction, sonner or later.

Developer
Sep 29, 2011 at 6:22 AM

Hi Kyle,

I'd like to create the Map Layers that work with IFeatureSource or IShapeSource. Could you share more details about your simplified map layer implementation? Is the source code available (on a branch or elsewhere)?

Especially, I'd like to know if you use IFeatureSource.Select() method or if you use IShapeSource.GetShapes() method in the custom layer.

Also, did you use the DrawRegions method to implement the custom drawing?

Thanks,

Jiri

 

Oct 12, 2011 at 11:28 AM

Kellison,

Can you send me the package for testing? I am evaluate this issue and will decide if I will stay with this library or not.

Thanks,

Chris

Developer
Oct 13, 2011 at 1:52 AM

Not sure what package you're referring to.  I think Jiri made some progress on an implementation for a LIDAR layer.

Kyle