EditMode vs IndexMode, etc

Developer
Dec 7, 2010 at 5:34 PM

After nearly two months of pretty much full time development with Dot spatial I find that more than half my time is spent debugging inconsistencies between Edit Mode/Index Mode    Features/ShapeIndices   FastDrawStates and Datarows, Vertices, etc

I understand that this is supposed to make DotSpatial blazingly fast.  And I know that for some applications this is of paramount importance.

However, there just has to be a better way.  All these parrallel arrays and mode switching based on dynamic internal state are just impossible for my limited mind to fathom.

Correct me if I'm wrong but it seems that the user base pushing for lightening fast access are probably not EVER editing these large data sets.  Maybe they even use a completely different tool to edit.  While, on the other hand, the user base that wants edit mode rarely loads a million features before editing... and if they do, they are prepared to wait a little. 

I'm in the latter category and this brings me to the question:

Given the current state of DotSatial and the apparent focus on high speed read-only access to Shape file format.  Is DotSpatial actually a suitable framework for Edit Mode applications.  Are there alternatives that have a more straightforward api for editing featuresets?

Mabey it is just me and my lack of understanding of the DotSpatial model?  Is anyone out there having good success writing full featured edting tools on DotSpatial?  Anywone care to post real world samples of DotSpatial Editing tools.

 

Thanks for your indulgence.

Garth

 

Developer
Dec 7, 2010 at 6:37 PM

You do raise some good points Garth.  It is very confusing to people and it is not the best model that we could have come up with if we had known that the NTS geometries would be unusable as rendering objects ahead of time.  That being said, it is not too late to create those simplified data objects, and there has been some discussion  an an initial effort at doing just that.  Tracking things exclusively by feature makes a lot of sense in the re-arrangeable in-ram case, but not much sense at all if you need to to load in a page at a time for the larger shapefiles.  We have a lot of ideas and desires to fulfill with this project, and there is a certain amount of inherent complexity that we are working with.  For instance, we are endeavoring to support external data providers which brings another large chunk of perhaps unnecessary complexity to the operation.

One thing we can look at doing is splitting up the FeatureSet.  Right now it is a blatent violation of the single responsibility principal.  It is at once a base class implementation for data access and shape type specific classes.  It is a wrapper class supporting internal featuersets for handling extensible IFeatureSet objects.  It (or its internal IFeatureSet) contain the Vertex array and PartIndices that are so critical for rendering quickly.  It has an optional, lazy instantiation of a list of "Features" which are easy to use but slower and take up more memory.  It has the DataTable, and also works as an IAttributeSource in case the data table is too large to fit in memory.  Mounted around it are extension methods to allow topologic funtions, reprojection and other things.

For developers just getting started it is useful because the literally should have to look no further than the FeatureSet to accomplish most of their vector tasks.  The disadvantage is that the different modes of operation and different behaviors under the hood lead to bugs and confusion.  One thing we have started looking at is the "FeatureRow/FeatureTable".  Instead of having a Feature with a DataRow and a problematic relationship there, we are looking at hosting editing designed where the DataRow IS the Feature in essence, and supports a "Geometry" and a "Shape" object on it for rendering or topology calculations.  It would support an FID style indexing system so that pages could be cycled for editing large datasets, while at the same time, each page would effectively be as easy to work with as a DatatTable, and have no extra weaknesses.  It would be inherently designed to store values in databases.  This would be ideal for editing or any form of manipulation that involves the attributes themselves directly.  The objects for this are created, but I don't yet have the code written that will populate a FeatureTable from a shapefile, or else render the content on a FeatureTable as a working layer.  I think this might still be ok for fast data access if we simply don't load the attribute rows or access the Geometry object on it, though even mostly empty it may take up more space than our existing vertex code.  We would lose the idea of "ordering" your features in the featureset all together, just as datarows have no inherent "order".  But we would gain from having a lot less ambiguity about how to use the objects.  External data providers will need to return FeatureTables with all the shapes or loaded with content from a specific page.  We may get clever and have a query involved here to, though that will depend on how easy it will be to implement it.  At least a spatial query should be supported.

So in a way, all your frustrations have been noted already, and there is a plan in place to come up with problem free alternatives, I just haven't finished implementing it.  Thanks for your feedback though because this is exactly what we need in order to turn around and say, you know what?  The existing solution isn't really ideal, and here is an example of a specific user post where this has been brought up as an issue.

Ted

 

 

Developer
Dec 7, 2010 at 7:03 PM
Edited Dec 7, 2010 at 7:58 PM

So, when can we have it?

I have, as mentioned above, lot's of cycles spent debugging the old method.  I'd rather be debugging (or creating) the new method.

How do I start now?

 

Garth

 

 

Developer
Dec 7, 2010 at 8:20 PM

So the next step is writing a reader that will populate the FeatureTable from a shapefile (or whatever data source you would prefer to use.)  There is an "IDataReader" interface that would probably make sense to support seeing as the DataTable should be able to use it directly, but another alternative is to take the open and save methods from our existing Line, Point, Polygon and Multipoint shapefile code and simply tweak them so that instead of creating a FeatureSet, the result is a FeatureTable.  Rather than using inheritance, I recommend we keep the FeatureTable as the in-ram representation only, and have a different class be responsible for reading the shapefile to create the FeatureTable.  Maybe a ShapefileReader or something.

I can start working on that tonight actually, but I can't do anything right now since I'm at work.  The step after that will be to restructure the FeatureLayer code slightly so that it works with FeatureTable instead.  These don't necessarily have to displace our existing code base, and so we can probably build this new skeleton around our existing one.  There are lots of structures that are currently set to work with FeatureLayers that will expect a FeatureSet at the moment, so I don't recommend simply adding a FeatureTable property to our existing FeatureLayer and attempting to put a bunch of conditional statements in.  It would make more sense to copy and paste the existing FeatureLayer content and trade out the FeatureSet, and then build up alternative forms as necessary to work with it.

Anyway, if you want to get started right away, why don't you take a first crack at porting the existing Shapefile reading code over into a new class that would be the "Shapefile reader".  Instead of a ShapeRange being added directly to a FeatureSet, embed the ShapeRange into a Shape object, add your new FeatureRow, and define the "Shape" property on the FeatureRow to be the newly created Shape.

If it's too much over the top, I will try to get the ball rolling tonight.  But if you are willing to sink some time, why don't you see what you can get together, and commit your changes before 5:00 PM Pacific time, and I will see what you've got and see what I can do to step it up to the next level.

Ted

 

 

 

Developer
Dec 7, 2010 at 10:39 PM

As a very easy preliminary step I commited the recently added OgrDataExtension project to the main line.  This extension exposes any OgrDataSource (provided it is supported by the currently installed OGR version) as an IDataReader.  Then I added a simple button to the right end of the Demo Map Tool Strip.  This button executes this code:

 

        private void toolStripButton4_Click_1(object sender, EventArgs e)
        {
            OpenFileDialog dlg = new OpenFileDialog();
            dlg.Filter = "ESRI Shape Files (*.shp)|*.shp";
            dlg.Title = "Select Shape File";

            if (dlg.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                IDataReader reader = new OgrDataReader(dlg.FileName);

                FeatureTable t = new FeatureTable();
                t.GeometryFactory = GeometryFactory.Default;
                t.Load(reader);

                string sMessage = String.Format("File opened. {0} features loaded", t.Rows.Count);
                MessageBox.Show(sMessage);
            }
        }

So, now we have a FeatureTable with rows, fids and Geometries read from a shape file.

 

What's next?  How do we get that into a map layer?

I've got 1:23 minutes left before my 5:00pm deadline - highly unusual   ;-) .   New challenge please.

Since I have extra time I'll see if I can get the shape file reader code mashed into an IDataReader implementation

 

 

 

Developer
Dec 7, 2010 at 11:34 PM

Fantastic =P.  This is great news.  We are well on our way now.  This relies on OGR, so similar to GDAL we need to make sure it is set up in a way that won't cause problems based on changing 32, 64.  I also need to create configurations that are designed for 3.5, so that our newly implemented GDAL libraries that are 4.0 specific can be turned off when we compile in 3.5.

Presuming all that is set up, we need to consider option A or B.

A) Find functionality that we would like to share between the different kinds of FeatureLayers, figure out an interface for it that we could retrofit onto both the IFeatureSet and the IFeatureTable, so they might both implement IFeatureBase.  Then we retool as much of our existing structure to use the IFeatureBase calls instead of what we currently use in order to handle the major tasks:

Rendering - e.g. IMapLayer implementation

Hosting Symbology, Symbolizer (and the associated dialogs for controling these)

Selection (probably based on FID in this case, but we already have ISelection and IndexSelection so we can probably use that).

B) Decide that there is no common ground or easy way to set up a common interface, and that it would be easier to retool everything to work with the new kind of layer.  For basic rendering we start with inheriting from Layer and implementing IMapLayer in order to do some custom drawing.  We could gradually build up funtionality on the new form of layer, possibly also creating the Polygon, Line, Point variants, and replica symbology dialogs.

A is potentially less long term work if we are smart about hooking up the interface, but might have compatibility headaches from people that try to cast to the FeatureSet case and don't get what they expect.  B requires a larger investment in copy-paste style creation of redundant classes and retooling how those redundant classes work.  B will be less error prone for anything that was designed with the expectation that the new layer classes exist, but I can foresee problems where existing code only tests for IFeatureLayer, IRasterLayer and IImageLayer, that won't be looking for our new layer types.  A lot of code uses IFeatureLayer in a way that expects IFeatureSet, but there are a lot of classes to make if we make new layers from scratch.  Either method may require significant work.  The advantage of B is that we won't break the existing code while we work on it.  A would require setting up new interfaces within our existing code, which is fine if we are careful, but the process is likely to break things.

 

Ted

 

Developer
Dec 7, 2010 at 11:34 PM

Hmmmm.....

Mashing [Polygon|Point|Line]ShapeFile reading logic into a DataReader looks more complicated than I'm willing to bite off at 4:25pm

 

I guess the existing Shapefile reading logic is probably much faster than OGR so it makes sense to try to preserve it.   Would you see this being done as readers are intended... "forward only row at a time"  or use the existing Shape file logic to load the entire thing to memory and then have then load the FeatureTable from there?

 

Developer
Dec 7, 2010 at 11:58 PM

Yes, I knew the 32/64 bit issues for OGR would need to be addressed.  If I knew how, I would have done it myself.  Can I leave that to you?

w.r.t A vs B I defer to your much larger vision.  Personally, I think the tendrils of Attribute table and Shapes will be found clutching to very deep bits of the existing code base.  I would lean toward starting anew with option B.

 

Garth

 

 

Developer
Dec 8, 2010 at 12:03 AM

The IDataReader is just one approach because it is something people might already be familiar with.  In practice, we just need a class that can be pointed to a shapefile and spit out a FeatureTable.  My feeling is the IDataReader isn't really designed to do that very well, and I don't think it even provides write access.  We have started a ShapeSource class that spits out shapes (possibly only polygons at the moment I can't remember) and an AttributeTable class that spits out the attributes in pages.  My thought to support this in .Net without OGR is to set up something that either relies on those or is similar in design, but can at the very least take a spatial extent to return a subset from the shapefile.  That is, it builds up a FeatureTable on the fly, but only for the shapes in a specified extent.  We could have an option on there to control whether just the shapes are loaded or whether the attributes are returned as well.  This class would not persist anything in memory, but would behave more like a data adapter to do the translations necessary.  After I set up the 3.5 configurations, I might have time tonight to make some progress in this area.  One trick we might look into for an intermediate layer case is to write adapter logic.  Create an IFeatureSet implementation that wraps a FeatureTable internally but as far as our layers are concerned for drawing or other things, this would work like a FeatureSet.  It just might have some things that are "unsupported" exceptions if necessary.  I think it would be ok as long as we sort of have it locked in index mode as far as our existing code base is concerned.  This might be easier than rewriting a the whole symbology layer architecture.

Developer
Dec 8, 2010 at 1:25 AM

"This relies on OGR, so similar to GDAL we need to make sure it is set up in a way that won't cause problems based on changing 32, 64.  I also need to create configurations that are designed for 3.5, so that our newly implemented GDAL libraries that are 4.0 specific can be turned off when we compile in 3.5"

Does this discussion refer to the GDAL/OGR dependencies in DotSpatial.Data.Rasters.Gdal1-7Extension? If not please disregard this post.

I used the VS2010 builds for the DotSpatial.Data.Rasters.Gdal1-7Extension but it should be pretty straight forward to use the other distributions as appropriate from http://vbkto.dyndns.org/sdk/  I often have difficulties in accessing this site not sure if it's an issue with the dynamic DNS service or the server behind it - would we consider mirroring it under DS?

Tamas discusses dependencies and other concerns http://szekerest.blogspot.com/2010/02/redistribution-of-shared-c-runtime.html

Developer
Dec 8, 2010 at 1:49 AM

Yes, the 1-7 versions are the ones that won't compile in 3.5.  The link didn't work for me just now.  I don't know that a mirror is necessary if we have the binaries as appropriate for our own build state.  That could be taken up with Dan or someone that is running one of the other servers.  It would probably not belong on the Codeplex site itself.  I am not sure that I will be successful setting up the configurations that I want completely.  Currently we can easily switch between Debug/Release and x86 and AnyCPU.  The configuration manager seems to support the creation of even more configurations however.  When I build with MSBuild, I can specify the 3.5 framework, which automatically invokes project settings in the project files for that case.  What I am not sure of is whether I can create a separate configuration for the configuration manager that would let us set it up so that certain projects don't build in 3.5.  In that case I think it would be fine to have the existing FWTools based GDAL build for 3.5 since the 1-7 code in a way depends on that extension existing anyway since it links to the source.  That was a great move btw.  It means that the 1-7 automatically got updated with some improvements I made to the other one.  The only thing we would be missing right now is 3.5 and x64 GDAL at the same time since the original GDAL reference was x86 only.

Ted

Developer
Dec 8, 2010 at 12:58 PM

This is an admittedly self-serving suggestion, but won’t the file-based approach we have been working on inherently solve many of these synchronization problems?

Kyle

From: shade1974 [mailto:notifications@codeplex.com]
Sent: Tuesday, December 07, 2010 1:38 PM
To: kellison@geocue.com
Subject: Re: EditMode vs IndexMode, etc [DotSpatial:237483]

From: shade1974

You do raise some good points Garth. It is very confusing to people and it is not the best model that we could have come up with if we had known that the NTS geometries would be unusable as rendering objects ahead of time. That being said, it is not too late to create those simplified data objects, and there has been some discussion an an initial effort at doing just that. Tracking things exclusively by feature makes a lot of sense in the re-arrangeable in-ram case, but not much sense at all if you need to to load in a page at a time for the larger shapefiles. We have a lot of ideas and desires to fulfill with this project, and there is a certain amount of inherent complexity that we are working with. For instance, we are endeavoring to support external data providers which brings another large chunk of perhaps unnecessary complexity to the operation.

One thing we can look at doing is splitting up the FeatureSet. Right now it is a blatent violation of the single responsibility principal. It is at once a base class implementation for data access and shape type specific classes. It is a wrapper class supporting internal featuersets for handling extensible IFeatureSet objects. It (or its internal IFeatureSet) contain the Vertex array and PartIndices that are so critical for rendering quickly. It has an optional, lazy instantiation of a list of "Features" which are easy to use but slower and take up more memory. It has the DataTable, and also works as an IAttributeSource in case the data table is too large to fit in memory. Mounted around it are extension methods to allow topologic funtions, reprojection and other things.

For developers just getting started it is useful because the literally should have to look no further than the FeatureSet to accomplish most of their vector tasks. The disadvantage is that the different modes of operation and different behaviors under the hood lead to bugs and confusion. One thing we have started looking at is the "FeatureRow/FeatureTable". Instead of having a Feature with a DataRow and a problematic relationship there, we are looking at hosting editing designed where the DataRow IS the Feature in essence, and supports a "Geometry" and a "Shape" object on it for rendering or topology calculations. It would support an FID style indexing system so that pages could be cycled for editing large datasets, while at the same time, each page would effectively be as easy to work with as a DatatTable, and have no extra weaknesses. It would be inherently designed to store values in databases. This would be ideal for editing or any form of manipulation that involves the attributes themselves directly. The objects for this are created, but I don't yet have the code written that will populate a FeatureTable from a shapefile, or else render the content on a FeatureTable as a working layer. I think this might still be ok for fast data access if we simply don't load the attribute rows or access the Geometry object on it, though even mostly empty it may take up more space than our existing vertex code. We would lose the idea of "ordering" your features in the featureset all together, just as datarows have no inherent "order". But we would gain from having a lot less ambiguity about how to use the objects. External data providers will need to return FeatureTables with all the shapes or loaded with content from a specific page. We may get clever and have a query involved here to, though that will depend on how easy it will be to implement it. At least a spatial query should be supported.

So in a way, all your frustrations have been noted already, and there is a plan in place to come up with problem free alternatives, I just haven't finished implementing it. Thanks for your feedback though because this is exactly what we need in order to turn around and say, you know what? The existing solution isn't really ideal, and here is an example of a specific user post where this has been brought up as an issue.

Ted

Read the full discussion online.

To add a post to this discussion, reply to this email (DotSpatial@discussions.codeplex.com@discussions.codeplex.com)

To start a new discussion for this project, email DotSpatial@discussions.codeplex.com@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

Developer
Dec 8, 2010 at 1:58 PM

Regarding the ShapeSource... just recently I contributed the point and line implementations of IShapeSource and IFeatureSource in addition to the polygon implementation that Ted did originally.  I am using those now, so they appear to be working.

Kyle

Developer
Dec 8, 2010 at 4:11 PM

Yes Kyle, this is an awesome contribution and will likely be involved as the vector providing access.  So certainly, those can be used  and at least the IShapeSource would not have any synchronization problems.  But I think that ultimately his concern was more directed at passing around in-ram content that was difficult for him to keep straight.  How can he edit the shapes and be assured that his edits were the ones that would be drawn, etc.  That is all the options on a FeatureSet like index mode, data rows on features that might not link up correctly, having internal IFeatureSet etc.  It can get quite confusing, mostly while doing editing.  I originally tried doing a concept where a Feature inherited from DataRow a long time ago when we first got started, but at the time the strong typed table base that makes it a possibility now didn't exist, and without the new auto-generated tables that are part of VisualStudio 2010, I would never have figured out that it was even possible.  Having learned it, I consider the new FeatureTable structure to be much more stable.  No more problems linking a "DataRow" property on a feature class to a potentially unrelated "DataTable" but it still remains to be seen if it works for fast loading.  If the DataRow doesn't slow it down too much then it probably wont hurt to set up some layer content that is designed to work with this.  The question would still exist of how do you populate these FeatureTables.  The answer may fall back to our IShapeSource under the hood.  We may also tinker with the IFeatureSource so that it has overloads that return FeatureTables in addition to FeatureSets.  My goal would to start with dual support.  That is, we don't necessarily do away with our current FeatureSet concept, but we simply add in the new kind of vector layers that work with FeatureTables.  The reality is that the FeatureTable is strictly an in ram representation and would not try to take on the second responsibility of being a data access portal to file or database sources.  So the FeatureTable does not really replace all the functionality of the current FeatureSet.  To support what we want in our final vision, the layer system would have to be able to work with either an IFeatureTable (say for features you create from scratch in memory) or an IFeatureSource.  What remains to be seen is whether we re-use the existing vector layers, create completely new layers and whether the new layers are even the same between the IFeatureSource and the FeatureTable variants.  I think the FeatureTable one might be set up as kind of a base class, and then the IFeatureSource layer might do some overriding where tables are loaded into memory and passed to methods on the base class one at a time so that we could have some code reuse.

 

Developer
Dec 8, 2010 at 4:30 PM

Hmmm.. So now we are talking about FeatureSets, FeatureSources and FeatureTables.  Are there examples of how to use a FeatureSource?

My objective is simple:

  • Open Shape File ( and preferably KML or other sources)
  • Insert Features
  • Delete Features
  • Edit Features
  • Save.

And have some confidence that

  1. my app won't crash somewhere in the process
  2. the output data is valid for the format.

 

It is entirely possible that this can be done with Dot Spatial as it is.  It's just that after two months I still don't see how.    Maybe all we need is some good sample code showing how this is meant to be done?  If FeatureSource fixes this problem it's fine with me.   I'm certainly not looking to rewrite a library that works.  I just would like to understand how I'm supposed to make DotSpatial work for data editing?

My original question seems to remain unanswered.  Is it just me?  Are there others out there having good success writing data edit apps with DotSpatial.  If so, please contribute sample code.

Garth

 

 

May 10, 2011 at 9:57 AM

Hey, have been reading your conversation. You two seem to have been working hard and doing alot. Please allow me to cut in here. I dont understand how the ISelection or Selection implementation works. Every time I use the selection tool to select a feature on the map (using either Mapwindow6 or Dotspacial), inside this event:

map1_SelectedIndexChange(obeject sender, EventArgs e)

{

[my code here:]

}

... I expect to get the index (or FID) of the selected feature. If the index I can use that to get the remaining attributes. Or, if I have the FID, I can use that to query an external database, which is my main goal. But the above event does not return any of that. Here is a snippet of what I wanna insert in the block above:

[my code here]:

MapWindow.Data.IFeature feat  = map1.GetFeatureLayers()[layer1].DataSet.GetFeature(index)

layer1 is layer am interested in.

index is the index of the selected feature, which the event does not seem to return.

I am stuck. Your kind help will be appreciated.... :)

 

Reagard

Dave

Developer
May 10, 2011 at 2:10 PM

Look at    FeatureLayer.Selection.    You'll find a list of selected objects and a set of methods to manipulate them.

May 12, 2011 at 12:51 PM

Whenever I on a feature on the map (or select a number of features) using the Selection tool in the toolbar, the SelectionChange event of the map gets fired, but then I see Count = 0 for the ISelection object. Nothing seems to be selected. Why is this. I have been looking into this for days now. Please help; and thanks for your previous help. Look forward to hearing from you...