File based Shapefile Layer for large shapefiles

Developer
Sep 29, 2011 at 5:19 AM

Hi DotSpatial developers,

I'm trying to implement a layer that will work with very large shapefiles or databases where the whole shapefile doesn't fit into memory. I'm quite confused by the current usage of EditMode/IndexMode and FastDrawnStates in FeatureSet.cs in DotSpatial. I didn't find any documenations or examples explaining the logic behing IndexMode and EditMode.

However, I found the IShapeSource and IFeatureSource interfaces that provide the capability of efficiently reading a sub-region (current extent bounding box) into memory without reading the whole big file.

When I look at the current MapPointLayer.cs, it has a method 'DrawRegions' that takes a list of bounding rectangles and draws all the points that are located within these rectangles. I'm deciding between two strategies:

A) Use IFeatureSource.Select method. Call IFeatureSource.Select in DrawRegions() passing the current map extent and maximum number of Feature objects that can fit into memory without occupying too much memory (? 10,000 Features ?). Then re-use the existing code in MapPointLayer to draw the FeatureSet that is returned by IFeatureSource.Select method. If there are more than 10,000 features within the current map extent, then do the drawing in 'chunks': run IFeatureSource.Select, which modifies the StartIndex, draw the result of IFeatureSource.Select to the 'backbuffer' bitmap, discard the result of IFeatureSource.Select, re-run IFeatureSource.Select with the new StartIndex, until all features within the view extent are drawn.

B) Use IShapeSource.GetShapes method in DrawRegions. Implement custom code that performs the drawing.

Any suggestions about using IShapeSource and IFeatureSource would be appreciated. Has anybody already created a layer that works with IShapeSource or IFeatureSource?

Thanks,

Jiri

Developer
Sep 30, 2011 at 12:12 AM

I made initial progress in creating a custom layer that implements IMapLayer. The key was implementing the DrawRegions() method.

In the code, DataSet is a custom data type (in my case: I'll use a LiDAR data type read using PDAL) that could use IShapeSource or IFeatureSource to load the data within the current view extent.  Right now, I'm randomly generating the points (1000 points each time) but I'm on a good path to wire up this code to IShapeSource and test it with a large shapefile / compare memory consumption.


public virtual void DrawRegions(MapArgs args, List<Extent> regions)
        {
            foreach (Extent boundingBox in regions)
            {
                Graphics g = args.Device ?? Graphics.FromImage(_backBuffer);
                Matrix origTransform = g.Transform;
                FeatureType featureType = FeatureType.Point;

                double minX = args.MinX;
                double maxY = args.MaxY;
                double dx = args.Dx;
                double dy = args.Dy;

                //reads the point array from the data source
                //DataSet implements or uses IShapeSource to read the points that are within the bounding box
                double[] vertices = DataSet.GetPointArray(boundingBox);

                //setup the point symbol
                Color randomColor = CreateRandomColor();
                Bitmap normalSymbol = CreateDefaultSymbol(randomColor, 4);

                //run the drawing operation
                int numPoints = vertices.Length / 2;

                for (int index = 0; index < numPoints; index++)
                {
                    Bitmap bmp = normalSymbol;

                    if (featureType == FeatureType.Point)
                    {
                        Point pt = new Point();
                        pt.X = Convert.ToInt32((vertices[index * 2] - minX) * dx);
                        pt.Y = Convert.ToInt32((maxY - vertices[index * 2 + 1]) * dy);

                        Matrix shift = origTransform.Clone();
                        shift.Translate(pt.X, pt.Y);
                        g.Transform = shift;

                        g.DrawImageUnscaled(bmp, -bmp.Width / 2, -bmp.Height / 2);
                    }
                }

                if (args.Device == null) g.Dispose();
                else g.Transform = origTransform;
            }
        }


Developer
Oct 3, 2011 at 5:32 PM
Edited Oct 3, 2011 at 5:37 PM
Greate! I'll use the framework to create wfs provider 
    //run the drawing operation
                int numPoints = vertices.Length / 2;

                for (int index = 0; index < numPoints; index++)
                {
                    Bitmap bmp = normalSymbol;

                    if (featureType == FeatureType.Point)
                    {
                        Point pt = new Point();
                        pt.X = Convert.ToInt32((vertices[index * 2] - minX) * dx);
                        pt.Y = Convert.ToInt32((maxY - vertices[index * 2 + 1]) * dy);

                        Matrix shift = origTransform.Clone();
                        shift.Translate(pt.X, pt.Y);
                        g.Transform = shift;

                        g.DrawImageUnscaled(bmp, -bmp.Width / 2, -bmp.Height / 2);
                    }
                }

==

If the numPoints is very large,should we should resample the features before draw them?

Oct 11, 2011 at 3:28 AM

Hi jiri,

have you had any solution to solve this problem?

lzz