Visibility and Importing- Repositories, Modules, Package, Files, Build Trees etc
Let start this section with a quote - "‘When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’" from Alice in Wonderland.
The reason is that in its evolution Go has changed it concepts around code packaging and how visibility and inclusion (import) are managed and like Humpty Dumpty is sometimes "to clever by half" so understanding its behaviour can be a point of frustraton.
The way Go behaves and the concepts behind this vary considerablely across:
- Go 1.11 / 1.12 - Introduced "modules" which provided a new dependency management systems that made version explicit. This meant that when creating a module you explicitly provide version information (Go uses a "semantic version" convention based on: Major, Minor and Patch release - i.e. v1.1.1) and when you import a module you explicity provide details of which version is acceptable (ie must be greater v1.2.1). To control if historical or new module behaviour as used the "GO111MODULE" environment variable was introduced (more on this later)
- Go 1.13 - Changed the the default behavior and management of "modules" based on enviromment variables: "GOPATH" and "GO111MODULE".
So Go's concepts and behaviour can be looked at pre-module and post-module.
NOTE: I am assuing that Go version is > 1.11 and so using the GO111MODULE environment variable, if you are using verson lower than this then GO111MODULE will have no impact and you will always get the "Pre-Module" behaviour.
Pre-module
Pre-module Go worked best if you structure your file systems source tree using Go convention:
project/ <=== workspace - gopath env should reference this bin goprog pkg <="==" go install puts library files here linux_amd64 first level is os_arch, so dependent on your org.url next reflects: from the src tree [owner ] reflecting: lib.a. reflecting lib package [.git] <<="==" optional: repository location #1 name space (import path) identifier (like "github.com") [.git]. #2 url above owner project respository prog for main goprog.go veryusefule.go use: import "lib veryuseful" helper.go. helper"< code>
The key Go structuring and visibility concepts where:
- Workspace - the OS file tree where Go source code and generated programs are. The source tree root should be defined via the GOPATH environment variable
- package - the collection of Go sources files within a single directory
- repository - the version control directory/s that could be part of content of source tree. Note that the .git respository placement is independent of Go and Go toes not interact with this.
- import path - this is the qualificaton path that it used when importing the package. This is not explicity defined "pre-module", but automtically determined based on where the package its in the directory structure relative to the "src/" directory that the package sits in.
To build your go project you should first go into the library tree and do:
- Pre-module library build / install - assuming Go 1.11 or greater: "GOPATH=<WORKSPACE> GO111MODULE=no go [build | install]"
Based on above tree example the library will have import path: "org.url[/owner]/lib", so ensure you have correct import path. Now build/install the local go program from "prog" directory (as per workshop above): "GOPATH=<WORKSPACE> GO111MODULE=no go [build | install]"
The buid /install process will place code in different locations relative to GOPATH based on package type:
- "main" package - will be put into <GOPATH>/bin
- other (library) packages - will be port into: <GOPATH>/pkg/<OS-ARCH>/<URL>[/<OWNER>]/<LIB>.a
You can see the symbols within your library ".a" (object archives> using "go tool nm <LIB>.a"
To get import your library packages into another Go program you use the "import" directive.
So with the structure above to import my "lib" I wold use: "import "org.url[\owner]\lib"
To invoke a function within this package I would have to qualify it with the package name to call one of the public visible function: " lib.PrintIt(s)". Note that the qualification only has to use the final package name and the function is upper case. Uppercase names are used by code to control visibility outside of the package.
So in go all source files within the same package have full visibility to all attributes / functions / classes within the same package (source tree directory).
Functions within source files that are in a different package only have visibilty to the "public" (upper case) declarations of another package.
Post-Module
So now post-module... where on top of above "pre-module" concepts Go introduces the "module".
- modules - which is a collection of related Go packages that are released together. Unlike the "packages" above which are delimited by the common source directory they reside, the modules is declared by the "go.mod" file within the package source directly and includes all the sub-directory below this unless this contains further "go.mod" file.
While the pre-module package import path was generated based on the position of the package directory relative to "src/" directory, for modues this is defined excplicity through the "go.mod" file within the package directory.
In additional to decoupling the naming from the position relative to "src/" introduction of modules also introduces explicit version management, which is also controlled via "go.mod" file.
So looking at source tree above, Post-Module this would have:
project/ <=== workspace - gopath env should reference this bin goprog pkg <="==" go install puts library files here linux_amd64 first level is os_arch, dependent on your org.url next reflects: from the src tree [owner ] reflecting: lib.a. reflecting lib package mod downloaded versioned copies of imported [.git] <<="==" optional: repository location #1 name space (import path) identifier (like "github.com") [.git]. #2 url above owner project respository prog for main goprog.go [go.mod] to control version modules include module veryusefule.go use: import "lib veryuseful" helper.go. helper" go.mod mandatory definition [vendor depenencies ...< code>
As per pre-module the behaviour can be controlled by the GO111MODULE environment variable but as of Go 1.15 the default behaviour is as per GO111MODULE=yes, so if you have pre-module packages without "go.mod" definition file, then this should be added or the control flag changed to maintain pre-module behavior.
The "vendor" directory contained cached code based on go.mod and this cached coded and version will take priority of the same package that might have been installed higher up in the hiearchy. The vendor directory cache was avialable "pre-module" as well, but was not controlled by the go.mod file (rather by invocation flags).
Visibillty of functions , attributes and classse is controlled by using upper case and with modules you can now have a hiearchy of packages all as part of the same module.
| Area | Notes |
| Compiling | Go does have a compiler, but generally everyting is done with the "go" command tool. This does much more that just compile the code ..
It does builds, install and in a simple golang only project makes "make" an unnecessary accessory
- build - does a compile check, but throws away result
- install - either builds a binary or library
|
| Libraries | Go has libraries, known is packages and modules but these are not like C/C++, where you have seperate header (.h) and generated object file (.o), which get collected into libaryes archive (.a) |
| Object Files | Go does not provide visible object files. Rather if you do an "go install" it will generate the binary executable or a go library (.a library) containing all functions/classes within your go package / module in single invocation |
| Environment Variables | Go uses a set of environment variable to control its behavior and where it places things. This is differnt to C/C++ where control is via -flags |
| Cache and Clearing it.. | The go tool is doing a lot work in the background. The problems is that things can get mixed up, with libraries not resolving.
A give away is when it it trying to find packages that are in your local code base, but is treating them like external packages, with the following error: blah blah blah
If you have done a clean, removed your ~/go directory and vendor generated diretory then you might need to reset the cache:
go clean -cache -modcache -i -r |
Go & Make
If Go automatically builds the binary then what is the role of Make ?
While Go automatically already does many things that would otherwise be managed by Make (or Ant in Java context), it only deal with go code and libraries. So if you have code that also has seperate sql, JavaScript, CSS etc then you will likely still need to use Make to pull all these other parts together as part of your overall build process.
Targets and Build Process
Lets cover basic program & library build process. Firstly some basic Make build target conventions. Generally standard Make build conventions define the following targets:
- build - builds the set of target: program, library in local build tree but do not install it for use
- clean - clean out all the generated files
- install - build and install the projects programs, libraries and documentation onto hosting machine for use
- uninstall - remove the build program, libraries and documentation from the machine (ie reverse install)
- dist - build distribution package for project which is targetted for deployment on another machine
Typically a Makefile will define the target programs and libraries and the source files that are used to build this. The dependency rules then define how to compile, build archives and link these together to create the executable. With make this is ussually done a file at a time (based on the rule) by make based on dependency rule.
This is different to Go where you run "go build" in a "main" package directory and it will automatically compile all the .go files in that directory (so I do not ask go to build a specific .go program, but will throw away the result.. which is why you use "go install" which does the build but keeps the result based the output directive (-o flag).
For a library package or module, you can go into the directory and it will automatically download all the dependencies (based on go.mod) and compile all the .go files.
So here are things that typical "make" and Go do differently.
- go build - throws away results, while "make build" keep result and aims to build the target
- go install - had behaviour more like "make build", but it acts on entire directories while make dependency rules typically results in actions on individual files
- go clean - has simillar behavior to a "make clean" target
- make install - does not have go equivalent, this would be something that you could still rely on make to do...
So to align Go and make working cooperatively together you need to have dependency rules in you Makefile tha take into account the Go behaviour.
For my "Commento" build Makefile I have top level Makefile that just contains the sub-directories as make targets.
For each of the Go sub-directory targets the build target has no dependencies, the result is that the build will always fire the "Go build / install / clean" command, which will operate on all the files within the package directory. So unlike a typical C/C++ Makefile, where make needs to be appear of the files in the directory, for the Go make, this is managed by Go and make just needs to know the packages/directories that is shoud enter and run the next Makefile for.
References & Links:
