> Zig does not, and will not, have VLAs in the language spec. Instead, you can allocate a slice on the heap. If you want to have the data on the stack, use an array as a bounded backing store, and work with a slice into it[.]
Too bad, aligned byte-typed VLAs (and a license to retype them as a struct) are what you need to get stack allocation across ABI boundaries the way Swift does it. (A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.) I guess I’ll have to use something else instead.
Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call.
At a fundamental level, runtime-known stack allocation harms code reusability.
Edit: commenters identified 2 more puzzle pieces below, but there's still one that didn't get asked about yet :P
A comptime_int-bounded alloca would achieve those goals, plus would be more space-efficient on average than the current strategy of always pessimistically allocating for the worst case scenario.
with the added bonus that if `count` is small, you can avoid splitting the stack around a big chunk of unused bytes. Don't underestimate the important of memory locality on modern CPUs.
instead of being namby pamby with the stack, it's simply better to take all of the desired maximimum. in zig, there's even provided a way to wrap it in an allocator so you can pretend like it's on the heap!
This conversation would benefit from using more rigorous technical terminology than "namby pampy". There is nothing namby pamby about allocating the precise amount of space that you need, and keeping your app's memory footprint optimized. That's called engineering.
real question: what are you going to do with the rest of the stack? are you in a situation where the stack and the heap might collide because you're that tight on resources? and let's say you take a function call that is about to overflow the stack. what should happen? error? panic? return null? silent fail?
there are no good choices in the case where you really need that thing you claim to need. recognizing that fact and picking different strategy is good engineering.
The first stack / heap collisions were not using VLA but fixed size arrays on the stack. Nowadays compilers do stack probing, which solves this problem also for VLAs. Yes, you get a segfault for stack overflow, but this has not much to do with VLAs or not, but with putting too much stuff on the stack. The thing is, VLAs allow you to reduce stack usage, by putting the right amount of stack on the stack and not a worst case.
The only downside is that they make it harder to control stack usage, but not a lot harder. So no, I do not think avoiding VLAs is good engineering.
This whole post is a strawman. I never said my reason was being tight on resources. Please reread the thread. Also don't forget that on modern architectures, the stack and heap can't "collide", because of guard pages.
> what are you going to do with the rest of the stack?
I'll leave it for the rest of the system. My app will use less memory, and since memory locality is improved, there will be fewer cache misses, meaning it runs faster too.
> let's say you take a function call that is about to overflow the stack
Stack overflows are impossible thanks to the comptime upper_bound parameter. That's the entire premise of this thread.
Zig does allow this, that's what GP is saying. You don't actually need to relocate your stack, you can just declare a portion of your stack (i.e. what would otherwise be the next N frames) to be the stack you'll use for recursion, and thereafter use that to recurse in to.
How does this work with interrupts, say in an embedded context, that execute on the current stack, and that may in some cases, be interrupted themselves. Do you add the maximum stack depth of all interrupt routines that could go off at the same time?
> Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call.
Sigh. So I have to choose between something I think might be useful, for something that too many languages have already soiled themselves with. Hopes that Zig has a better solution, but not optimistic.
Our stack compels me to work in Swift, Kotlin, Elixir, and Python. I use the async feature of Swift and Kotlin when some library forces me to. I actually preferred working with GCD before Swift had to join the async crowd. Elixir of course just has this problem solved already.
I frequently ask others who work in these languages how often they themselves reach for the async abilities of their languages, and the best I ever get from the more adventurous type is “I did a play thing to experiment with what I could do with it”.
RE: Elixir I have a feeling that the zig's i/o strategy will enable me to bring back the zig-async-dependent yielding nifs in zigler. I'm really hopeful io interface will have a yield() function, that would be even better!
All Zig code is in one compilation unit, so the compiler has access to the entire function call graph. Cycles in the graph (recursion) cause an error. To break cycles in the graph, one must use a language builtin to call a function using a different stack (probably obtained via heap allocation).
Does this mean it's impossible in Zig to do strictly Stack related recursion and just by the mere inclusion of a recursive function your implicitly getting heap allocations alongside?
You can put a big buffer on the stack, and use this buffer to break your cycles. At some point you'll run out of this buffer and be forced to handle failure, rather than triggering a stack overflow segfault.
So it will be the same thing but with more (error handling) steps.
This annoyance can be avoided by avoiding recursion. Where recursion is useful, it can be done, you just have to handle failure properly, and then you'll have safety against stack overflow.
Wait, so how do I write mutually recursive functions, say for a parser? Do I have to manually do the recursion myself, and stick everything in one big uber-function?
I see. So in some sense the actual unit of compilation is smaller and units can be combined or "linked" even if compiled at different times on different machines?
Zig's linker will calculate this information automatically in most cases when statically linking (via analysis of machine code disassembly). Otherwise, there is a default upper bound stack value, overridable via user annotation.
> Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call.
That’s a genuinely interesting point. I don’t think known sizes for locals are a hard requirement here, though threading this needle in a lower-level fashion than Swift would need some subtle language design.
Fundamentally, what you want to do is construct an (inevitably) runtime-sized type (the coroutine) out of (by problem statement) runtime-sized pieces (the activation frames, itself composed out of individual, possibly runtime-sized locals). It’s true that you can’t then allow the activations to perform arbitrary allocas. You can, however, allow them to do allocas whose sizes (and alignments) are known at the time the coroutine is constructed, with some bookkeeping burden morally equivalent to maintaining a frame pointer, which seems fair. (In Swift terms, you can construct a generic type if you know what type arguments are passed to it.) And that’s enough to have a local of type of unknown size pulled in from a dynamic library, for example.
Again, I’m not sure how a language could express this constraint on allocas without being Swift (and hiding the whole thing from the user completely) or C (and forcing the user to maintain the frames by hand), so thank you for drawing my attention to this question. But I’m not ready to give up on it just yet.
> At a fundamental level, runtime-known stack allocation harms code reusability.
This is an assertion, not an argument, so it doesn’t really have any points I could respond to. I guess my view is this: there are programs that can be written with alloca and can’t be written without (unless you introduce a fully general allocator, which brings fragmentation problems, or a parallel stack, which is silly but was in fact used to implement alloca historically). One other example I can give in addition to locals of dynamically-linked types is a bytecode interpreter that allocates virtual frames on the host stack. So I guess that’s the other side of being opinionated—those whose opinions don’t match are turned away.
Frankly, I don’t even know why I’m defending alloca this hard. I’m not actually happy with the status quo of just yoloing a hopefully maybe sufficiently large stack. I guess the sticking point is that you seem to think alloca is obviously the wrong thing, when it’s not even close to obvious to me what the right thing is.
Alloca is a fundmentally insecure way of doing allocations. Languages that promote alloca will find themselves stuck in a morass of security messes and buffer overflows. If Zig were to adopt alloca, it would make the catastrophic mistake that plagued C for over several decades and introduce permanently unfixable security issues for another generation of programming languages.
I’ve been defending alloca() here, but no, strdupa() (not to be confused with shlwapi!StrDupA on Windows) is a bad idea. In cases that I think are acceptable, the size of the allocation is reasonably small and does not come from outside the program. Here you’re duplicating a string that you probably got somewhere else and don’t really control. That means you don’t really know if and when you’re going to overflow the stack, which is not a good position to be in.
(Once upon a time, MSLU, a Microsoft-provided Unicode compatibility layer for Windows 9x, used stack-allocated buffers to convert strings from WTF-16 to the current 8-bit encoding. That was also a bad idea.)
> A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.
Was kind of the other way around, given the whole OS/2 versus Windows history, and that COM started as the evolution of OLE and VBX technologies, Windows 9X and Windows NT weren't as COM heavy as OS/2 was with SOM.
There was no COM to worry about on Windows 3.x back in 1991.
Also SOM was so much better, bettwen C++, Smalltalk and C, with support for meta-classes and proper inheritance implementation across such disparate languages.
I think there may be room to expand this implementation to support such a use case. Right now it enforces an `.auto` layout of the struct provided in order to ensure alignment, but its easy to imagine supporting an `extern struct` with a defined layout.
Conceivably, an implementation of this `ResizableStruct` that uses an array buffer as backing rather than a heap allocation, and supports the defined layout of an extern struct, could be used to work across the ABI
you can certainly allocate on the stack (like alloca). you just have to overallocate a compile-time known size and have some sort of fallback mechanism or fail if the size requested exceeds the amount created.
moreover since the stack allocator is just an allocator, you can use it with any std (or user) datastructure that takes an allocator.
I am wondering if this is more of an unclearly defined memory ownership problem rather than a problem of what types you have to use to interact with C ABIs or FFI calls.
I mean you could also just abstract the allocation away and handle it after the function pointer to your bridge, right?
I thought about dynamically sized types (DSTs) in zig recently. Was considering writing about it. I came to a different conclusion. Why not use zig's opaque?
It's pretty clean at this imo:
Less metaprogramming but I think nicer to use in some cases.
I would describe this approach as 'intrusive' - you're storing the lengths of the arrays behind the pointer, enforcing a certain layout of the memory being allocated.
Because the solution outlined in the article stores the lengths alongside the pointer, instead of behind it, there is room for it to work across an ABI (though it currently does not). It's more like a slice in this way.
You could in theory implement your opaque approach using this as a utility to avoid the headache of alignment calculations. For this reason, I think that makes the approach outlined in the article more suitable as a candidate for inclusion in the standard library.
Yeah I think mine is more about being able to provide a `host()` helper function instead of a `.get(.host)` meta function. It is somewhat boilerplate-y. I think it's really a matter of taste haha. Likely yours would be useful regardless if this is done a lot, since it abstracts some of it, if one wants that.
I've entertained further expanding this API to expose a comptime generated struct of pointers. From the Connection use-case detailed in the article, it would look something like this:
The opaque keyword in Zig doesn't support method definitions - it creates an incomplete type that must be implemented elsewhere, not a type that can have inline methods and fields like your example attempts.
> opaque {} declares a new type with an unknown (but non-zero) size and alignment. It can contain declarations the same as structs, unions, and enums
So it can contain methods, just like structs. The only thing it cannot contain is fields (which the example above doesn't contain). An opaque type can only be used as a pointer, it's like a `typedef void* ...` in C but with the possibility to add declarations within a namespace
I sincerely mean this when I write this: please make more Girls on the Beach albums. I really love Splif Tape, reminds me of Julie Ruin during that timeframe as well (2015ish).
oh, man! you made my day with this comment. those days are long behind me. we were young, dumb, and inebriated. if you're looking for the bands we were trying to sound like, there was a resurgence of this surf/girl group sound in the early 2010s. look for bands like Shannon and the Clams, Hunx and His Punx, Nobunny, Harlem, Ty Segall, Thee Oh Sees.. basically anyone on the now defunct Burger Records
Because arrays simply do not deal with fragmentation. Yes, you could probaly get decent performance on a modern system that has memory overcommit strategy where you could allocate sparse adress ranges where you would probaly never run out of pointers unless you actually write to your variable array.
But its just kind of mediocre and you're better off actually dealing with the stack if you can actually deal with certain fixed sizes.
array-like storage with dynamic size has existed since forever - it's vector. over or undercommitting is a solved problem
VLA is the way to bring that into type system, so that it can be it's own variable or struct member, with compiler auto-magic-ing size reading to access members after it
> auto-magic-ing size reading to access members after it
From the article
>we now have everything we need to calculate the size, offset and alignment of every field, regardless of their positioning in the struct.
>init to allocate the memory
>get to get a pointer to a field
>resize to resize the arrays
>deinit to free the memory
You're now suggesting to do exactly what the article is about without being aware of it.
Should I try Zig? I wanted to learn something that's more low level than the JVM/Node and i have been contemplating rust and go. zig never occurred to me until now.
I think it's simpler than rust, the basics are easier to learn and will get you a long way before you need the more advanced stuff.
Go might be easier because it's garbage collected. If you want something closer to C, but not C itself, Zig is a good choice.
Zig articles tend to get a little too excited about rediscovering longstanding techniques.
The author has described a metaprogramming utility for allocating a contiguous hunk of memory, carving this hunk into fields (in the article's example, a fixed-sized Client header, then some number of bytes for host, then some number of bytes for read_buffer, and then some for write_buffer). I'll acknowledge the syntax is convenient, but
That first pointer is needless indirection and probably a cache miss. You should (unless you have specific performance data showing otherwise) store the sizes in the object header, not in an obese pointer to it. (It's bigger than even a fat pointer.)
You have raised a good discussion point or two, but I am not inclined to engage with them due to the tone you've created with the rest of your comment.
Would it be productive to jump into a thread on a Ruby article and puff your feathers about how you've always been able to do this in Perl, and also in Python 4 you can do XYZ? I don't think so.
For whatever reason, inevitably in threads on systems languages, someone comes in and postures themselves defensively like this. You might want to reflect on that.
> I think you're misinterpreting OP's excitement here. The technique isn't novel but the ergonomics are.
This, of course, but I also consciously made the decision to write for an audience less familiar with the language and concepts being discussed. That excitement can translate to engaging a less disgruntled and more curious reader than the OP.
While C++ isn't without warts, and WG21 does indeed a good job adding a few more, many time people aren't able to do ergonomic stuff in C++, because they insist in using it like C with Classes.
> Zig does not, and will not, have VLAs in the language spec. Instead, you can allocate a slice on the heap. If you want to have the data on the stack, use an array as a bounded backing store, and work with a slice into it[.]
Too bad, aligned byte-typed VLAs (and a license to retype them as a struct) are what you need to get stack allocation across ABI boundaries the way Swift does it. (A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.) I guess I’ll have to use something else instead.
Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call.
At a fundamental level, runtime-known stack allocation harms code reusability.
Edit: commenters identified 2 more puzzle pieces below, but there's still one that didn't get asked about yet :P
A comptime_int-bounded alloca would achieve those goals, plus would be more space-efficient on average than the current strategy of always pessimistically allocating for the worst case scenario.
with the added bonus that if `count` is small, you can avoid splitting the stack around a big chunk of unused bytes. Don't underestimate the important of memory locality on modern CPUs.2017: https://github.com/ziglang/zig/issues/225
when I had been only thinking about zig for 2 years, I thought the same.
I'd be curious if you expanded your reasoning, your comments in that thread never explained anything for me.
> It's too tempting to use incorrectly.
A compile-time-determined upper bound would solve this.
> The stack is allocated based on a compile-time-determined upper bound.
A compile-time-determined upper bound would solve this too.
Shouldn't a performance-oriented language give the programmer tools to improve memory locality? And what's wrong with spexguy's idea?
instead of being namby pamby with the stack, it's simply better to take all of the desired maximimum. in zig, there's even provided a way to wrap it in an allocator so you can pretend like it's on the heap!
This conversation would benefit from using more rigorous technical terminology than "namby pampy". There is nothing namby pamby about allocating the precise amount of space that you need, and keeping your app's memory footprint optimized. That's called engineering.
real question: what are you going to do with the rest of the stack? are you in a situation where the stack and the heap might collide because you're that tight on resources? and let's say you take a function call that is about to overflow the stack. what should happen? error? panic? return null? silent fail?
there are no good choices in the case where you really need that thing you claim to need. recognizing that fact and picking different strategy is good engineering.
The first stack / heap collisions were not using VLA but fixed size arrays on the stack. Nowadays compilers do stack probing, which solves this problem also for VLAs. Yes, you get a segfault for stack overflow, but this has not much to do with VLAs or not, but with putting too much stuff on the stack. The thing is, VLAs allow you to reduce stack usage, by putting the right amount of stack on the stack and not a worst case. The only downside is that they make it harder to control stack usage, but not a lot harder. So no, I do not think avoiding VLAs is good engineering.
This whole post is a strawman. I never said my reason was being tight on resources. Please reread the thread. Also don't forget that on modern architectures, the stack and heap can't "collide", because of guard pages.
> what are you going to do with the rest of the stack?
I'll leave it for the rest of the system. My app will use less memory, and since memory locality is improved, there will be fewer cache misses, meaning it runs faster too.
> let's say you take a function call that is about to overflow the stack
Stack overflows are impossible thanks to the comptime upper_bound parameter. That's the entire premise of this thread.
Yes, let's be "namby Pamby" with the cache lines storing the hot part of the stack, that sounds like an awesome idea!
I thought Zig was all about maximum performance. Sometimes I just want a little bit of stack memory, which will often already be in L1 cache.
Zig does allow this, that's what GP is saying. You don't actually need to relocate your stack, you can just declare a portion of your stack (i.e. what would otherwise be the next N frames) to be the stack you'll use for recursion, and thereafter use that to recurse in to.
How does this work with interrupts, say in an embedded context, that execute on the current stack, and that may in some cases, be interrupted themselves. Do you add the maximum stack depth of all interrupt routines that could go off at the same time?
> Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call.
Sigh. So I have to choose between something I think might be useful, for something that too many languages have already soiled themselves with. Hopes that Zig has a better solution, but not optimistic.
Our stack compels me to work in Swift, Kotlin, Elixir, and Python. I use the async feature of Swift and Kotlin when some library forces me to. I actually preferred working with GCD before Swift had to join the async crowd. Elixir of course just has this problem solved already.
I frequently ask others who work in these languages how often they themselves reach for the async abilities of their languages, and the best I ever get from the more adventurous type is “I did a play thing to experiment with what I could do with it”.
RE: Elixir I have a feeling that the zig's i/o strategy will enable me to bring back the zig-async-dependent yielding nifs in zigler. I'm really hopeful io interface will have a yield() function, that would be even better!
https://www.youtube.com/watch?v=lDfjdGva3NE&t=1819s
Love the excitement
> there's still one that didn't get asked about yet :P
C libraries?
Or function pointers (especially given that Zig's been moving towards encouraging vtables over static dispatch)?
Bingo. That completes the picture: https://github.com/ziglang/zig/issues/23367
How does this work given... recursion?
Even on languages without VLAs one can implement a simulacra of them with recursion.
All Zig code is in one compilation unit, so the compiler has access to the entire function call graph. Cycles in the graph (recursion) cause an error. To break cycles in the graph, one must use a language builtin to call a function using a different stack (probably obtained via heap allocation).
Does this mean it's impossible in Zig to do strictly Stack related recursion and just by the mere inclusion of a recursive function your implicitly getting heap allocations alongside?
You can put a big buffer on the stack, and use this buffer to break your cycles. At some point you'll run out of this buffer and be forced to handle failure, rather than triggering a stack overflow segfault.
So it will be the same thing but with more (error handling) steps.
This annoyance can be avoided by avoiding recursion. Where recursion is useful, it can be done, you just have to handle failure properly, and then you'll have safety against stack overflow.
Wait, so how do I write mutually recursive functions, say for a parser? Do I have to manually do the recursion myself, and stick everything in one big uber-function?
Does Zig offer (guaranteed) tail call optimisation?
> Where recursion is useful, [...]
Recursion is so useful, most imperative languages even have special syntax constructs very specific special cases of recursion they call 'loops'.
> Does Zig offer (guaranteed) tail call optimisation?
Yes[1]. You can use the @call builtin with the .always_tail modifier.
[1]: https://ziglang.org/documentation/master/#call> All Zig code is in one compilation unit
How do incremental compilation and distributed compilation work?
Subtrees should be cacheable and parallelizable?
Single compilation unit does not imply that the results of the compilation of the different parts of that unit cannot be cached.
I see. So in some sense the actual unit of compilation is smaller and units can be combined or "linked" even if compiled at different times on different machines?
what about extern functions?
Zig's linker will calculate this information automatically in most cases when statically linking (via analysis of machine code disassembly). Otherwise, there is a default upper bound stack value, overridable via user annotation.
https://github.com/ziglang/zig/issues/1006
> Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call.
That’s a genuinely interesting point. I don’t think known sizes for locals are a hard requirement here, though threading this needle in a lower-level fashion than Swift would need some subtle language design.
Fundamentally, what you want to do is construct an (inevitably) runtime-sized type (the coroutine) out of (by problem statement) runtime-sized pieces (the activation frames, itself composed out of individual, possibly runtime-sized locals). It’s true that you can’t then allow the activations to perform arbitrary allocas. You can, however, allow them to do allocas whose sizes (and alignments) are known at the time the coroutine is constructed, with some bookkeeping burden morally equivalent to maintaining a frame pointer, which seems fair. (In Swift terms, you can construct a generic type if you know what type arguments are passed to it.) And that’s enough to have a local of type of unknown size pulled in from a dynamic library, for example.
Again, I’m not sure how a language could express this constraint on allocas without being Swift (and hiding the whole thing from the user completely) or C (and forcing the user to maintain the frames by hand), so thank you for drawing my attention to this question. But I’m not ready to give up on it just yet.
> At a fundamental level, runtime-known stack allocation harms code reusability.
This is an assertion, not an argument, so it doesn’t really have any points I could respond to. I guess my view is this: there are programs that can be written with alloca and can’t be written without (unless you introduce a fully general allocator, which brings fragmentation problems, or a parallel stack, which is silly but was in fact used to implement alloca historically). One other example I can give in addition to locals of dynamically-linked types is a bytecode interpreter that allocates virtual frames on the host stack. So I guess that’s the other side of being opinionated—those whose opinions don’t match are turned away.
Frankly, I don’t even know why I’m defending alloca this hard. I’m not actually happy with the status quo of just yoloing a hopefully maybe sufficiently large stack. I guess the sticking point is that you seem to think alloca is obviously the wrong thing, when it’s not even close to obvious to me what the right thing is.
Alloca is a fundmentally insecure way of doing allocations. Languages that promote alloca will find themselves stuck in a morass of security messes and buffer overflows. If Zig were to adopt alloca, it would make the catastrophic mistake that plagued C for over several decades and introduce permanently unfixable security issues for another generation of programming languages.
Any thoughts on the use of strdupa()? I do not use it, but I wonder if that is dangerous too, considering it uses alloca().
I’ve been defending alloca() here, but no, strdupa() (not to be confused with shlwapi!StrDupA on Windows) is a bad idea. In cases that I think are acceptable, the size of the allocation is reasonably small and does not come from outside the program. Here you’re duplicating a string that you probably got somewhere else and don’t really control. That means you don’t really know if and when you’re going to overflow the stack, which is not a good position to be in.
(Once upon a time, MSLU, a Microsoft-provided Unicode compatibility layer for Windows 9x, used stack-allocated buffers to convert strings from WTF-16 to the current 8-bit encoding. That was also a bad idea.)
I don't have anything against alloca(), but then again, I don't use it at all. I stick to malloc() / free(), and in case of strings, asprintf().
Didn't stop rust from using it internally.
I don't think Rust uses alloca internally for anything. You may be thinking of Swift, which I think uses alloca for ABI shenanigans.
How does it do that?
I don’t know why you’re downvoted, alloca is a mistake.
Out of curiosity, why is knowing the size of locals required, exactly? Because it avoids dynamic allocations for each fiber?
Does anything stop a user from doing this with inline assembly?
Wisdom
> A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.
Was kind of the other way around, given the whole OS/2 versus Windows history, and that COM started as the evolution of OLE and VBX technologies, Windows 9X and Windows NT weren't as COM heavy as OS/2 was with SOM.
There was no COM to worry about on Windows 3.x back in 1991.
https://www.edm2.com/index.php/SOM_%26_DSOM_-_An_Introductio...
Also SOM was so much better, bettwen C++, Smalltalk and C, with support for meta-classes and proper inheritance implementation across such disparate languages.
I think there may be room to expand this implementation to support such a use case. Right now it enforces an `.auto` layout of the struct provided in order to ensure alignment, but its easy to imagine supporting an `extern struct` with a defined layout.
Conceivably, an implementation of this `ResizableStruct` that uses an array buffer as backing rather than a heap allocation, and supports the defined layout of an extern struct, could be used to work across the ABI
you can certainly allocate on the stack (like alloca). you just have to overallocate a compile-time known size and have some sort of fallback mechanism or fail if the size requested exceeds the amount created.
moreover since the stack allocator is just an allocator, you can use it with any std (or user) datastructure that takes an allocator.
I am wondering if this is more of an unclearly defined memory ownership problem rather than a problem of what types you have to use to interact with C ABIs or FFI calls.
I mean you could also just abstract the allocation away and handle it after the function pointer to your bridge, right?
I thought about dynamically sized types (DSTs) in zig recently. Was considering writing about it. I came to a different conclusion. Why not use zig's opaque?
It's pretty clean at this imo: Less metaprogramming but I think nicer to use in some cases.
going off memory so I expect it to not actually compile, but I've definitely done something like this before.I would describe this approach as 'intrusive' - you're storing the lengths of the arrays behind the pointer, enforcing a certain layout of the memory being allocated.
Because the solution outlined in the article stores the lengths alongside the pointer, instead of behind it, there is room for it to work across an ABI (though it currently does not). It's more like a slice in this way.
You could in theory implement your opaque approach using this as a utility to avoid the headache of alignment calculations. For this reason, I think that makes the approach outlined in the article more suitable as a candidate for inclusion in the standard library.
Yeah I think mine is more about being able to provide a `host()` helper function instead of a `.get(.host)` meta function. It is somewhat boilerplate-y. I think it's really a matter of taste haha. Likely yours would be useful regardless if this is done a lot, since it abstracts some of it, if one wants that.
I've entertained further expanding this API to expose a comptime generated struct of pointers. From the Connection use-case detailed in the article, it would look something like this:
I haven't done this because I'm not yet convinced it's worth the added complexityThe opaque keyword in Zig doesn't support method definitions - it creates an incomplete type that must be implemented elsewhere, not a type that can have inline methods and fields like your example attempts.
https://ziglang.org/documentation/master/#opaque
> opaque {} declares a new type with an unknown (but non-zero) size and alignment. It can contain declarations the same as structs, unions, and enums
So it can contain methods, just like structs. The only thing it cannot contain is fields (which the example above doesn't contain). An opaque type can only be used as a pointer, it's like a `typedef void* ...` in C but with the possibility to add declarations within a namespace
Edit: the documentation doesn't contain any example of an opaque type which happens to have declarations inside. But here is one: https://codeberg.org/andrewrk/daw/src/commit/38d3f0513bf9bfc...
I am the author of this post, let me know if you have any questions or feedback :)
I sincerely mean this when I write this: please make more Girls on the Beach albums. I really love Splif Tape, reminds me of Julie Ruin during that timeframe as well (2015ish).
oh, man! you made my day with this comment. those days are long behind me. we were young, dumb, and inebriated. if you're looking for the bands we were trying to sound like, there was a resurgence of this surf/girl group sound in the early 2010s. look for bands like Shannon and the Clams, Hunx and His Punx, Nobunny, Harlem, Ty Segall, Thee Oh Sees.. basically anyone on the now defunct Burger Records
[flagged]
one thing I never understood about VLAs - discussion about them always hits a "can't put it on stack safely" and gets halted, forever
why not to make it heap-only type? it seems such a useful addition to type system, why ignore it due to one usecase?
Because arrays simply do not deal with fragmentation. Yes, you could probaly get decent performance on a modern system that has memory overcommit strategy where you could allocate sparse adress ranges where you would probaly never run out of pointers unless you actually write to your variable array.
But its just kind of mediocre and you're better off actually dealing with the stack if you can actually deal with certain fixed sizes.
...what are you talking about?
array-like storage with dynamic size has existed since forever - it's vector. over or undercommitting is a solved problem
VLA is the way to bring that into type system, so that it can be it's own variable or struct member, with compiler auto-magic-ing size reading to access members after it
> auto-magic-ing size reading to access members after it
From the article
>we now have everything we need to calculate the size, offset and alignment of every field, regardless of their positioning in the struct. >init to allocate the memory >get to get a pointer to a field >resize to resize the arrays >deinit to free the memory
You're now suggesting to do exactly what the article is about without being aware of it.
Those effectively exist. They're called slices.
You can also put them safely on the stack. The VLA is discussion is just irrational.
Should I try Zig? I wanted to learn something that's more low level than the JVM/Node and i have been contemplating rust and go. zig never occurred to me until now.
I think it's simpler than rust, the basics are easier to learn and will get you a long way before you need the more advanced stuff. Go might be easier because it's garbage collected. If you want something closer to C, but not C itself, Zig is a good choice.
The author is literally proposing to implement arrays of variant types
Zig articles tend to get a little too excited about rediscovering longstanding techniques.
The author has described a metaprogramming utility for allocating a contiguous hunk of memory, carving this hunk into fields (in the article's example, a fixed-sized Client header, then some number of bytes for host, then some number of bytes for read_buffer, and then some for write_buffer). I'll acknowledge the syntax is convenient, but
1. we've done this since time immemorial in C. See https://learn.microsoft.com/en-us/windows/win32/api/evntcons...
2. you can implement this pattern ergonomically in C++, and even moreso once the C++26 reflection stuff comes online
3. the zig implementation is inefficient. It desugars to
That first pointer is needless indirection and probably a cache miss. You should (unless you have specific performance data showing otherwise) store the sizes in the object header, not in an obese pointer to it. (It's bigger than even a fat pointer.)You have raised a good discussion point or two, but I am not inclined to engage with them due to the tone you've created with the rest of your comment.
Would it be productive to jump into a thread on a Ruby article and puff your feathers about how you've always been able to do this in Perl, and also in Python 4 you can do XYZ? I don't think so.
For whatever reason, inevitably in threads on systems languages, someone comes in and postures themselves defensively like this. You might want to reflect on that.
[flagged]
I assure you, I am not the one who appears to be reacting in a hair trigger hypersensitive manner in this thread :)
I think you're misinterpreting OP's excitement here. The technique isn't novel but the ergonomics are.
> you can implement this pattern ergonomically in C++,
lmao
> I think you're misinterpreting OP's excitement here. The technique isn't novel but the ergonomics are.
This, of course, but I also consciously made the decision to write for an audience less familiar with the language and concepts being discussed. That excitement can translate to engaging a less disgruntled and more curious reader than the OP.
While C++ isn't without warts, and WG21 does indeed a good job adding a few more, many time people aren't able to do ergonomic stuff in C++, because they insist in using it like C with Classes.