Thursday 10 December 2020

Building CReW Generate Dummy Variables

Background:

This story goes back to the 1980's when I was asked to test new bank software and to document what errors I uncovered.  I completed an Error Incident Report and walked that form to the next building for review by the development manager.  Finding errors meant that I got to leave my windowless cubicle and walk in the California sunshine (at least temporarily).  The developers needed to know what process they would need to perform to recreate the incident and what the expected results of such actions should be.  Upon return to my desk, I would have to summarize the days findings and explain back to the project management what issues were found and what their potential impacts would be on the bank's production environment if left unresolved.

The tried and true waterfall development process left testing until after the requirements and design were completed and I eagerly awaited software to find what were sure to be plenty of bugs.  The biggest contention was whether the "expected" results were well defined in the requirements because I would report "errors" that could be considered as missed requirements.

Creating a CReW macro does not include requirement documents, design specifications, use cases, or an approval process.  In my, Mark's, POV I create what I think captures the essence of a design pattern (assembly of Alteryx tools) into a single tool.  Some of these patterns are fairly simple and straight-forward and reduce the time to configure multiple-tools and some patterns are difficult for beginners to gain the concepts for.  In either case, the tool obfuscates the pattern and makes something easier for the user.

As an Alteryx ACE, I'm able to attend developer days where we have an opportunity to share feedback with the development (sometimes with product managers) team and discuss our views and needs as well as to give our feedback on their ideas.  Observant as I am to the NDA, that's where that story ends and this macro blog begins.

Let's talk Cross Tab"

Who doesn't like a Cross Tab tool?  Frankly, I don't know why it is a "Cross Tab" tool and isn't a "Crosstab" tool.  There are more than 200 ideas in the Alteryx Community returned when you query that term where 1/3 of these ideas are authored by Alteryx ACEs.  Just because there are literally hundreds of ideas posted, doesn't mean that the tool is broken.  It means that improvements are requested.  The cross tab tool is at the heart of the create dummy variables process and this process was the topic of my latest rant to Alteryx.

Cross Tab - Areas of potential improvement

Here are a few of my personal grievances:
  • Configuration is not saved as some new tools do
  • Dropdown and select are not searchable
  • Domain/headers replace non-alphanumeric characters with underscore
  • Output columns are in alphabetical order
  • You need data that supports all possible values otherwise Output columns are not generated
  • Downstream tools fail if not run with sufficient data

Cross Tab - Work-arounds

  • Use a find replace tool to prepend a sort sequence to each field value (to change the incoming data) so that the column name sorts data (e.g. 00_Sunday, 01_Monday).  Later I can dynamically rename the columns
  • Use an Ensure Fields (CReW) macro to create needed columns of data (after Cross Tab). 
  • Place a select in front of the cross tab.  Only bring a minimum amount of fields into cross tab.  Join back on either record sequence or recordid depending on need (e.g. use of AMP or if output records aren't 1:1 to input)
  • Sample the output data (0 records) and save to YXDB. This saves the metadata for a good output.  You can later Union this data within the workflow and ensure order and presence of data 

Let's talk Dummy Variables

After my short (not so) rant, I realized that I should take on the challenge to improve Alteryx myself through the creation of the Generate Dummy Variables tool.  It won't improve the Cross Tab, but does address the issues that take the greatest amount of my time when creating workflows that either model or score data.  With 30 or more community posts on the subject, maybe it will help folks in the same boat as I am in.



Defining Scope:

Since I don't apply requirement documents or use cases to my CReW activities, I need to set boundaries for functionality and time to build.  What can I create in a day?  Yes, when I build a macro I hope to have it created in a day.  That day generally spans for more than a week, but that includes blogs, builds, testing, feedback, documentation and icon selection.  

Scope for Dummy Variables:

The scope is for a single variable (despite my knowledge that you may have multiple source fields to convert).  I want to accept categorical data elements and provide the ability to rename the output columns to include the column name.  If possible, I want to add the functionality of an ENSURE FIELDS macro (CReW), but not require the user to have that macro downloaded.

Scope Creep:

That idea to include the variable name as the first part of the output column name (e.g. Color_G) created a thought that a custom prefix might be preferred to the actual field name.  Then I ran into problems with spaces (thank you Cross Tab) in the output column name field.  When I added the ensure fields capability (to pretend that data values not present were accounted for), I had space problems and NULL value issues that needed more creativity.  Finally, I was ready for testing and went to fellow ACE, Dan Languedoc for testing support.  He suggested (insisted) that I allow for INT fields to be supported as categorical variables.  "Fine!", I'll add them.  He later said that my macro was indestructible.

Development:

I was unhappy with my final product.  I felt that it wasn't easy to walk-through to explain to others and didn't feel satisfied with the macro.  I had added functionality and had to ensure compliance in multiple locations of the workflow.  It was not pretty.  So I salvaged what I could and re-wrote the macro.  It took me about an hour to come up with the current version.

Testing:

While I was building the macro, I was actively testing the macro in another workflow.  I was using different configuration settings as well as different incoming data (e.g. things with and without spaces).  When I want to see what's happening inside of the macro, I use a "debug" trick to place a Macro Output tool before and/or after a tool and look at the results in the secondary workflow.  I can see what the data looks like going into and out of a dynamic rename or other tools that are in the middle of the workflow and can quickly reconnect them to another point of interest.

Defensive Configuration or "APA Readiness":

Will this macro fail in the future?  I hope not.  I hope that it works and that I never have to maintain or support it again.  But what might be worse is to have consistently wrong results.  The biggest risk that I saw was dropping or muliplying records.  I use a TEST tool to ensure that the macro input record count exactly matches the output record count.  If you drop a record (e.g. Create Points) and don't let the user know about it, I think that it is an error.  I also added a check that if you ask to ENSURE field valuess that the macro will error if you don't actually input a field value.  Also, if you copy/paste a field value into the ensured fields, if you don't mark it as required, that we will ignore the request.  

Now if the macro fails, we can all blame Dan.  He said that it was good to go.

P.S.:  Here's a link to the macro!

36 comments:

  1. You need to have a good network marketing strategy, using the Internet to provide online marketing tool. With Internet marketing company, will help you to get your website in search engine advertising and optimization services provided by many of the results, but eventually you need to determine what kind of network marketing will give you the best of a product line. Leadgeneration

    ReplyDelete
  2. There are a number of quality construction career training possibilities that allow students to train for a career in construction. Students looking to pursue the necessary education to work in construction can do so by enrolling in an accredited vocational or trade school. Students can choose to specialize in a number of areas. Students can train for certificates or degree. Degrees range from an associates to a masters level. آرین سازه

    ReplyDelete
  3. They will thusly educate you with respect to whatever your task should hold fast to in the Building Regulations. XMX

    ReplyDelete
  4. Assuming you have an endorsed examiner giving the Building Control Service, the duty regarding ensuring that Building Regulations are conformed to is theirs.https://matterhorn-wholesale.com/

    ReplyDelete
  5. except if you're building reviewer is 130 years of age the consolidated insight of the organization is of little advantage. The experience and information on the individual building and additionally bug monitor nearby is what matters.https://buildingrenovation4u.co.uk/

    ReplyDelete
  6. When starting a job or start a new business, there are many difficulties encountered. Try to be persistent and achievements will bring good results to those who try.

    ReplyDelete
  7. Hello! Today there are many great applications and software. If you follow someone now, it's really easy with today's developed technology. In the world of technology, there are a variety of applications, it will be difficult for you to choose, here, will help you solve, give advantages and disadvantages, suitable for your needs.

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. Probate real estate investing provides an opportunity for estate administrators to sell real estate holdings. This is particularly beneficial for administrators who are struggling to pay mortgage payments or maintain upkeep on property held in probate. pif finance

    ReplyDelete
  10. Antistatic corrugated box for packaging Item Antistatic corrugated box Material Paperboard, corrugated Color Black MOQ 3,000pcs FOB Price USD 0.1-1.0/pcs Surface disposal Embossing Hot stamping, glossing/matte lamination, varnishing, glitter, creasing, die cutting Delivery time 10-15days. Visit our website anti static box to learn more.

    ReplyDelete
  11. This site has particular software articles which emit an impression of being a significant and significant for you individual, able software installation. This is the spot you can get help for any software installation, usage, and crack.
    ez cd audio converter Crack
    wondershare tidymymusic Crack
    cyberghost vpn Crack
    macro recorder Crack

    ReplyDelete
  12. There are a number of quality construction career training possibilities that allow students to train for a career in construction. Students looking to pursue the necessary education to work in construction can do so by enrolling in an accredited vocational or trade school. Students can choose to specialize in a number of areas. Students can train for certificates or degree. Degrees range from an associates to a masters level. الارشاد السياحي

    ReplyDelete
  13. What does a baby spend a lot of the first 12 months of their life doing? Sleeping. And that sleep will almost certainly happen on the crib mattress. In truth, your child will most likely sleep on his or her crib mattress till round four years of age. So buying a crib mattress is a very important buy, and right here you will find tips how to decide on the proper one. Best beds

    ReplyDelete
  14. The OSHA 10 hour construction course is required in the states of New York, Nevada, Missouri, Massachusetts, New Hampshire, Connecticut, and Rhode Island for work on publicly funded projects of varying amounts. Additionally, the states of Nevada and Connecticut require the course to be retaken every 5 years. In virtually all cases, the online version of the OSHA 10 hour construction course meets these requirements and is a fast, efficient way to obtain the training needed. Towing Company Lake Charles LA

    ReplyDelete
  15. Plumbing is one of the professions that are highly rewarding and evergreen as the plumbing system is an essential part of every home, whether new or old. A plumbing system is meant for bringing and distributing freshwater in the home for all needs, and for draining the wastewater out of the home. The quality of plumbing system is also important for the durability of the building as a bad drainage or leaking water can damage the building quite before its useful life as per design. salt lake city plumber

    ReplyDelete
  16. Calling out a plumbing contractor for an emergency plumbing issue is most often a costly exercise. This article discusses the most common emergency plumbing issues that you are likely to encounter as a home-owner. plumber west jordan utah

    ReplyDelete
  17. Why pay for a plumber when you can learn to do many plumbing tasks yourself! One of the most important functions of a home or property is its plumbing system. Without plumbing, a home would be a mere shell. For people interested in pursuing plumbing for basic knowledge or as a means of work, plumbing courses are readily available to help you get started. bathroom renovation near me

    ReplyDelete
  18. The boundary conditions of a framed plumbing graph detail a specific diffeomorphism across the S X S euler number differential. The boundary framing on M, where M is the graph manifold variant, is modified by framed graph manifold classifications along the euler number weight vertex. To make sense of the plumbing graph modifications, the vertices of the plumbing must be defined from degrees 0 to 1. plumbers bountiful utah

    ReplyDelete
  19. You will often find a more specific 2-year program in "plumbing design" at a number of colleges. This shorter degree program will definitely get you what you need. Still you will want to gain experience in your field prior to graduation to give you better chances once you are ready to start your career. plumbing draper utah

    ReplyDelete
  20. Plumbing is one of the professions that are highly rewarding and evergreen as the plumbing system is an essential part of every home, whether new or old. A plumbing system is meant for bringing and distributing freshwater in the home for all needs, and for draining the wastewater out of the home. The quality of plumbing system is also important for the durability of the building as a bad drainage or leaking water can damage the building quite before its useful life as per design. plumber riverton ut

    ReplyDelete
  21. There are a number of quality construction career training possibilities that allow students to train for a career in construction. Students looking to pursue the necessary education to work in construction can do so by enrolling in an accredited vocational or trade school. Students can choose to specialize in a number of areas. Students can train for certificates or degree. Degrees range from an associates to a masters level. stamped concrete houstontexas

    ReplyDelete
  22. Working with a construction staffing agency is simple, saving you time and money by reducing your overall costs. Find out how easy it is to find reliable, skilled labor when you trust a construction staffing agency to supply a workforce of tradesmen for your construction project and boost your bottom line while you're at it. general contractor San Jose CA

    ReplyDelete
  23. Construction in Maryland has reached a high peak in recent years as compared to other states. While Indiana has 3.1%, Pennsylvania 2.3% and New York only 1.4% growth in construction jobs, Maryland construction jobs have increased by about 3.7% in the period between February and March 2010. general contractor San Jose CA

    ReplyDelete
  24. Crane accidents, fire explosions, roof cave-ins, heavy equipment fall, trench collapse, electrocution and even scaffolding breakage. What do all these things have in common? They are just but some of the most common types of construction site accidents occurring every year in every construction site that result to the construction workers' personal injuries or even worse, death. Here are more of these life-threatening situations that you may want to watch out for. Georgia and surrounding areas

    ReplyDelete
  25. Needless to say, the construction industry is badly broken and needs fixing. How does the industry rise up and meet the challenges of customer demand for higher quality, improved profitability, and the shortage of skilled workers? Lean thinking is a new way to manage construction. concrete repair lansing mi

    ReplyDelete
  26. A tax official must be suspicious when is noticed that sale prices for surface area of construction projects are massively in the thresholds established by VAT Law. According to this law, if the amounts of surface areas sold at this minimum price level are constructor's own product or part of the asset inventory of a taxable person's economic activity, they should not exceed the limit of private use, which includes normal family consumption, other employed persons, relatives and friend of taxable person. new homes in lakewood ranch fl

    ReplyDelete
  27. There are many different types of construction projects. How do you categorize them, and know what type of project a particular element of construction is? Here are seven different categories for construction projects. realtor fort mill sc

    ReplyDelete
  28. There are various sorts of fittings accessible on the lookout and you ought to pick just those that are suggested for your pipes installations and pipeline designs.Pipe Relining Picnic Point
    Kitchen Plumbing Picnic Point

    ReplyDelete
  29. Glad to be one of many visitants on this awing web site : D. Deck Repair

    ReplyDelete
  30. Thanks for taking the time to discuss this topic. I really appreciate it. I’ll stick a link of this entry in my blog. handyman services kalamazoo mi

    ReplyDelete
  31. There are some fascinating points in time in this article but I don’t know if I see all of them heart to heart. There may be some validity but I’ll take hold opinion till I look into it further. Good article , thanks and we would like more! Added to FeedBurner as properly Visit our site

    ReplyDelete
  32. Trades are always in demand and plumbers are no exception. There are steps to take in order to achieve a plumbing license. In this article we will go over how to get a plumbing apprenticeship. Cranbourne Landscaping

    ReplyDelete
  33. Garden and landscape lighting is often overlooked or ends up being just the bland wash of a security floodlight and maybe some ugly, low-level spreadlights illuminating paths. https://gardenprofy.com/

    ReplyDelete
  34. Emergency care centers are quick and convenient places for quick medical checkups. These clinics provide treatment for injuries and illnesses that do not require hospitalization or surgery. Many health plans provide nurse lines, but nurse lines can be difficult to call for specific diagnoses. An emergency care center will probably ask you to come for an assessment, and give you an estimate of the waiting time. A registered nurse will examine you and determine if you need medication or IV. Read the full info here urgent care.

    ReplyDelete