more disucssion of retag redundancy check

[web.git] / personal / _drafts / stacked-borrows-implementation.md

diff --git a/personal/_drafts/stacked-borrows-implementation.md b/personal/_drafts/stacked-borrows-implementation.md
index b4035fea626c6aa54d73da6baa4dfdedf90a6e57..5d7a7e70364a30488879f46a70902dede9285bc6 100644 (file)
--- a/personal/_drafts/stacked-borrows-implementation.md
+++ b/personal/_drafts/stacked-borrows-implementation.md
@@ -34,9 +34,8 @@ now.
 
 Ready?  Let's get started.  I hope you brought some time, because this is a
 rather lengthy post.  If you are not interested in a detailed description of
 
 Ready?  Let's get started.  I hope you brought some time, because this is a
 rather lengthy post.  If you are not interested in a detailed description of

-Stacked Borrows, you can skip most of this post and go right to
-[section 4](#4-differences-to-the-original-proposal).  If you only want to know
-how to help, jump to [section 6](#6-evaluation-and-how-you-can-help).
+Stacked Borrows, you can skip most of this post and go right to [section 4].  If
+you only want to know how to help, jump to [section 6].

 
 ## 1 Enforcing Uniqueness
 
 
 ## 1 Enforcing Uniqueness
 


@@ -325,6 +324,7 @@ fn demo2() {
 {% endhighlight %}
 
 ### 2.2 Dereferencing a Pointer
 {% endhighlight %}
 
 ### 2.2 Dereferencing a Pointer

+[section 2.2]: #22-dereferencing-a-pointer

 
 As we have seen, we consider the tag of a pointer already when dereferencing it,
 before any memory access happens.  The operation on a dereference never mutates
 
 As we have seen, we consider the tag of a pointer already when dereferencing it,
 before any memory access happens.  The operation on a dereference never mutates


@@ -359,6 +359,7 @@ The following checks are done on every pointer dereference:
       their stack (same check as if the tag had no timestamp).
 
 ### 2.3 Accessing Memory
       their stack (same check as if the tag had no timestamp).
 
 ### 2.3 Accessing Memory

+[section 2.3]: #23-accessing-memory

 
 On an actual memory access, we know the tag of the pointer that was used to
 access (unless it was a raw pointer, in which case the tag we see is
 
 On an actual memory access, we know the tag of the pointer that was used to
 access (unless it was a raw pointer, in which case the tag we see is


@@ -439,8 +440,10 @@ references are copied: At the beginning of every function, all inputs of
 reference type get retagged.  On every assignment, if the assigned value is of
 reference type, it gets retagged.  Moreover, we do this even when the reference
 value is inside the field of a `struct` or `enum`, to make sure we really cover
 reference type get retagged.  On every assignment, if the assigned value is of
 reference type, it gets retagged.  Moreover, we do this even when the reference
 value is inside the field of a `struct` or `enum`, to make sure we really cover

-all references.  However, we do *not* descend recursively through references:
-Retagging a `&mut &mut u8` will only retag the *outer* reference.
+all references.  (This recursive descend is already implemented, but the
+implementation has not landed yet.)  However, we do *not* descend recursively
+through references: Retagging a `&mut &mut u8` will only retag the *outer*
+reference.

 
 Retagging is the *only* operation that generates fresh tags.  Taking a reference
 simply forwards the tag of the pointer we are basing this reference on.
 
 Retagging is the *only* operation that generates fresh tags.  Taking a reference
 simply forwards the tag of the pointer we are basing this reference on.


@@ -470,8 +473,8 @@ fn demo0() {
 }
 {% endhighlight %}
 
 }
 {% endhighlight %}
 

-For each reference, `Retag` does the following (with one special exception that
-will be explained later):
+For each reference, `Retag` does the following (we will slightly refine these
+instructions later):

 
 1. Compute a fresh tag, `Uniq(_)` for a mutable reference and `Shr(Some(_))` for
    a shared reference.
 
 1. Compute a fresh tag, `Uniq(_)` for a mutable reference and `Shr(Some(_))` for
    a shared reference.


@@ -490,6 +493,9 @@ will be explained later):
       2. If the location is outside of `UnsafeCell`, it gets frozen with the
          timestamp of the new reference.
 
       2. If the location is outside of `UnsafeCell`, it gets frozen with the
          timestamp of the new reference.
 

+One high-level way to think about retagging is that it computes a fresh tag, and
+then performs a reborrow of the old reference with the new tag.
+

 ### 3.2 When Pointers Escape
 
 Creating a shared reference is not the only way to share a location: We can also
 ### 3.2 When Pointers Escape
 
 Creating a shared reference is not the only way to share a location: We can also


@@ -598,42 +604,63 @@ on locations inside `UnsafeCell`.  In fact, it is entirely equivalent to
 but we have to accept that there might be active mutable references assuming
 exclusive access to the same locations.  That on its own is not enough, though.
 
 but we have to accept that there might be active mutable references assuming
 exclusive access to the same locations.  That on its own is not enough, though.
 

-I also added a special exception: When "creating a raw reference"
-(`EscapeToRaw`, or `Retag` on a shared reference when we are inside an
-`UnsafeCell`), we still first check that the pointer this reference is created
-one is allowed to be dereferenced -- and we remember the position of the item on
-the stack that justifies this, let's call this `ptr_idx`.  Remember, that does
-not change the stack, so it has no adverse effect in `demo_refcell`. Next, we do
-a very peculiar check:
-
-1.  Determine if the new reference we are creating (it will have a `Shr` tag) is
-    *already dereferencable*.  This is possible, for example, if the location
-    already got shared previously.  This is also the case in `demo_refcell`:
-    Creating `rc_shr` pushed a `Shr` onto the stack.  We again remember the
-    position of the item that justifies this, let's call it `new_idx`.
-
-    - If this is the case, we moreover *compare* the `new_idx` and `ptr_idx`.
-      If `new_idx` is *above* `ptr_idx` on the stack, that means that not only
-      is the location already being shared (so we can dereference with our new
-      `Shr` tag), moreover *the pointer we create this reference from has
-      already been shared*.  The new pointer is already "derived from" the one
-      this reference is based on.  Basically, everything that we wanted to do
-      while creating this reference has already happened.  In this case, we just
-      do nothing else: We leave the stacks unchanged.
-
-    - Otherwise, we continue as usual.
-
-This rule has the effect that in our example above, when we create `shr_ref`, we
-do not pop `Uniq(2)` off the stack, and hence the access through `mut_ref` at
-the end remains valid.
-
-This may sound like a weird special case, and it is.  I would surely not have
-thought of this if `RefCell` would not force our hands here.  However, it also
-does not seem to break any of the important properties of the model (mutable
-references being unique and shared references being read-only except for
-`UnsafeCell`).
+I also added a new check to the retagging procedure: Before taking any action
+(i.e., before step 3, which could pop items off the stack), we check if the
+reborrow is redundant: If the new reference we want to create is already
+dereferencable (because it item is already on the stack and, if applicable, the
+stack is already frozen), *and* if the item that justifies this is moreover
+"derived from" the item that corresponds to the old reference, then we just do
+nothing.  Here, "derived from" means "further up the stack".  Basically, the
+reborrow has already happened and the new reference is ready for use, and
+(because of that "derived from" check), we know that using the new reference
+will *not* pop the item corresponding to the old reference off the stack.  In
+that case, we avoid popping anything, to keep other references valid.
+
+This rule applies in our example above when we create `shr_ref` from `mut_ref`.
+There is already a `Shr` on the stack (so the new reference is dereferencable),
+and the item matching the old reference (`Uniq(0)`) is below that `Shr` (so
+after using the new reference, the old one remains dereferencable).  Hence we do
+nothing, keeping the `Uniq(2)` on the stack, such that the access through
+`mut_ref` at the end remains valid.
+
+This may sound like a weird rule, and it is.  I would surely not have thought of
+this if `RefCell` would not force our hands here.  However, as we shall see in
+[section 5], it also does not to break any of the important properties of the
+model (mutable references being unique and shared references being read-only
+except for `UnsafeCell`).  Moreover, when pushing an item to the stack (at the
+end of the retag action), we can be sure that the stack is not yet frozen: If it
+was frozen, the reborrow would be redundant.
+
+With this extension, the instructions for retagging and `EscapeToRaw` now look
+as follows:
+
+1. Compute a fresh tag: `Uniq(_)` for a mutable reference, `Shr(Some(_))` for a
+   shared reference, `Shr(None)` if this is `EscapeToRaw`.
+2. Perform the checks that would also happen when we dereference this reference.
+   Remember the position of the item matching the tag in the stack.
+3. Redundancy check: If the new tag passes the checks performed on a
+   dereference, and if the item that makes this check succeed is *above* the one
+   we remembered in step 2 (where the "frozen" state is considered above every
+   item in the stack), then we stop.  We are done for this location.
+4. Perform the actions that would also happen when an actual access happens
+   through this reference (for shared references a read access, for mutable
+   references a write access).
+   Now the location cannot be frozen any more: If the fresh tag is `Uniq`, we
+   just unfroze, if the fresh tag is `Shr` and the location was already frozen
+   then the redundancy check (step 3) would have kicked in.
+5. If the new tag is `Uniq`, push it onto the stack.
+6. If the new tag is `Shr`, push a `Shr` item to the stack.  Then, if the
+   location is outside of `UnsafeCell`, it gets frozen with the timestamp of the
+   new reference.
+
+The one thing I find slightly unsatisfying about the redundancy check is that it
+seems to overlap a bit with the rule that on a *read* access, a `Shr` item
+matches a `Uniq` tag.  Both of these together enable the read-only use of
+mutable references that have already been shared; I would prefer to have a
+single condition enabling that instead of two working together.

 
 ## 4 Differences to the Original Proposal
 
 ## 4 Differences to the Original Proposal

+[section 4]: #4-differences-to-the-original-proposal

 
 The key differences to the original proposal is that the check performed on a
 dereference, and the check performed on an access, are not the same check.  This
 
 The key differences to the original proposal is that the check performed on a
 dereference, and the check performed on an access, are not the same check.  This


@@ -641,7 +668,10 @@ means there are more "moving parts" in the model, but it also means we do not
 need a weird special exception for `demo1` any more like the original proposal
 did.  The main reason for this, however, is that on an access, we just do not
 know if we are inside an `UnsafeCell` or not, so we cannot do all the checks we
 need a weird special exception for `demo1` any more like the original proposal
 did.  The main reason for this, however, is that on an access, we just do not
 know if we are inside an `UnsafeCell` or not, so we cannot do all the checks we

-would like to do.
+would like to do.  Accordingly, I also rearranged terminology a bit.  There is
+no longer one "reactivation" action, instead there is a "deref" check and an
+"access" action, as described above in sections [2.2][section 2.2] and
+[2.3][section 2.3].

 
 Beyond that, I made the behavior of shared references and raw pointers more
 uniform.  This helped to fix test failures around `iter_mut` on slices, which
 
 Beyond that, I made the behavior of shared references and raw pointers more
 uniform.  This helped to fix test failures around `iter_mut` on slices, which


@@ -676,13 +706,15 @@ reference never gets used again we do not care much about enforcing any
 guarantees for it.  (This is another example of a coincidental fix, where I had
 a surprisingly passing test case and then investigated what happened.)
 
 guarantees for it.  (This is another example of a coincidental fix, where I had
 a surprisingly passing test case and then investigated what happened.)
 

-And finally, there is this special exception about creating raw references, to
-fix `demo_refcell`.
+The redundancy check during retagging can be seen as refining a similar check
+that the original model did whenever a new reference was created (where we
+wouldn't change the state if the new borrow is already active).

 
 Finally, the notion of "function barriers" from the original Stacked Borrows has
 not been implemented yet.  This is the next item on my todo list.
 
 ## 5 Key Properties
 
 Finally, the notion of "function barriers" from the original Stacked Borrows has
 not been implemented yet.  This is the next item on my todo list.
 
 ## 5 Key Properties

+[section 5]: #5-key-properties

 
 Let us look at the two key properties that I set out as design goals, and see
 how the model guarantees that they hold true in all valid (UB-free) executions.
 
 Let us look at the two key properties that I set out as design goals, and see
 how the model guarantees that they hold true in all valid (UB-free) executions.


@@ -712,12 +744,14 @@ fn demo_mut_unique(our: &mut i32) -> i32 {
 {% endhighlight %}
 
 The proof sketch goes as follows: After retagging the reference, we know it is
 {% endhighlight %}
 
 The proof sketch goes as follows: After retagging the reference, we know it is

-at the top of the stack and the location is not frozen.  For any access
-performed by the unknown code, we know that access cannot use the tag of our
-reference because the tags are unique and not forgeable.  Hence if the unknown
-code accesses our locations, that would pop our tag from the stack.  When we use
-our reference again, we know it is on the stack, and hence has not been popped
-off.  Thus there cannot have been an access from the unknown code.
+at the top of the stack and the location is not frozen.  (The "redundant
+reborrow" rule does not apply because a fresh `Uniq` tag can never be
+redundant.)  For any access performed by the unknown code, we know that access
+cannot use the tag of our reference because the tags are unique and not
+forgeable.  Hence if the unknown code accesses our locations, that would pop our
+tag from the stack.  When we use our reference again, we know it is on the
+stack, and hence has not been popped off.  Thus there cannot have been an access
+from the unknown code.

 
 Actually this theorem applies *any time* we have a reference whose tag we can be
 sure has not been leaked to anyone else, and which points to locations which
 
 Actually this theorem applies *any time* we have a reference whose tag we can be
 sure has not been leaked to anyone else, and which points to locations which


@@ -765,13 +799,13 @@ fn demo_shr_frozen(our: &u8) -> u8 {
 {% endhighlight %}
 
 The proof sketch goes as follows: After retagging the reference, we know the
 {% endhighlight %}
 
 The proof sketch goes as follows: After retagging the reference, we know the

-location is frozen.  If the unknown code does any write, we know this will
-unfreeze the location.  The location might get re-frozen, but only at the
-then-current timestamp.  When we do our read after coming back from the unknown
-code, this checks that the location is frozen *at least* since the timestamp
-given in its tag, so if the location is unfrozen or got re-frozen by the unknown
-code, the check would fail.  Thus the unknown code cannot have written to the
-location.
+location is frozen (this is the case even if the "redundant reborrow" rule
+applies).  If the unknown code does any write, we know this will unfreeze the
+location.  The location might get re-frozen, but only at the then-current
+timestamp.  When we do our read after coming back from the unknown code, this
+checks that the location is frozen *at least* since the timestamp given in its
+tag, so if the location is unfrozen or got re-frozen by the unknown code, the
+check would fail.  Thus the unknown code cannot have written to the location.

 
 One interesting observation here for both of these proofs is that all we rely on
 when the unknown code is executed are the actions performed on every memory
 
 One interesting observation here for both of these proofs is that all we rely on
 when the unknown code is executed are the actions performed on every memory


@@ -785,13 +819,14 @@ but it is nevertheless interesting and might be useful in a "lower-level MIR"
 that does not permit dereferences in paths.
 
 ## 6 Evaluation, and How You Can Help
 that does not permit dereferences in paths.
 
 ## 6 Evaluation, and How You Can Help

+[section 6]: #6-evaluation-and-how-you-can-help

 
 I have implemented both the validity invariant and the model as described above
 in miri. This [uncovered](https://github.com/rust-lang/rust/issues/54908) two
 [issues](https://github.com/rust-lang/rust/issues/54957) in the standard
 library, but both were related to validity invariants, not Stacked Borrows.
 With these exceptions, the model passes the entire test suite.  There were some
 
 I have implemented both the validity invariant and the model as described above
 in miri. This [uncovered](https://github.com/rust-lang/rust/issues/54908) two
 [issues](https://github.com/rust-lang/rust/issues/54957) in the standard
 library, but both were related to validity invariants, not Stacked Borrows.
 With these exceptions, the model passes the entire test suite.  There were some

-more test failures in earlier versions (as mentioned in section 4), but the
+more test failures in earlier versions (as mentioned in [section 4]), but the

 final model accepts all the code covered by miri's test suite.  Moreover I wrote
 a bunch of compile-fail tests to make sure the model catches various violations
 of the key properties it should ensure.
 final model accepts all the code covered by miri's test suite.  Moreover I wrote
 a bunch of compile-fail tests to make sure the model catches various violations
 of the key properties it should ensure.